Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.

Q1: The effectiveness of the proposed VPG-C has not been fully validated. On the proposed DEMON benchmark, the improvement of VPG-C compared to InstructBLIP is quite limited, which does not align with the expectation that completed details would significantly enhance VPG. The improvement on the MME benchmark is also not significant.

A1: Thank you for your valuable feedback. We are grateful for the opportunity to further clarify and emphasize the unique strengths of our proposed VPG-C method.

(1.1) Data and Parameter Efficiency. Our method requires fine-tuning only 6.3M parameters with 64K synthetic instruction data, which is significantly more efficient than InstructBLIP's approach of fine-tuning 107M parameters using 16M human-annotated instruction data [Xu et al., 2023].

(1.2) Similarity in InstructBLIP's Training Data and Benchmark Tasks. The InstructBLIP's training datasets share considerable similarity with tasks in the DEMON and MME benchmarks. For instance, the OCR-VQA dataset used in InstructBLIP is also employed in the DEMON benchmark. Further, tasks like "Visual Conversation," "OKVQA," and "Complex Reasoning" in InstructBLIP closely resemble "Multi-Modal Dialogue," "Knowledge Grounded QA," and "Multi-Image Reasoning" in the DEMON benchmark. A recent study [Liu et al., 2023] has also indicated that the type of instruction tuning data used in InstructBLIP can significantly enhance performance on benchmarks like MME, marking a considerable improvement from 502.8 to 1377.6.

(1.3) Further Experiments Inspired by Your Suggestions. Motivated by your insightful suggestions, we conduct additional experiments to explore the scaling effect of instruction tuning data. We utilize a subset of InstructBLIP’s training data, including OKVQA, A-OKVQA, OCRVQA, TextCaps, and VQA-v2. As shown in the following tables, we find that even with a partial set of InstructBLIP's training data, the diversity and high quality of these manually annotated datasets substantially enhance the performance of VPG-C, notably surpassing InstructBLIP.

(* indicates fine-tuning with InstructBLIP's training data)

We are grateful for your suggestions, which have significantly contributed to improving our paper! We have incorporated the results of these additional experiments into our revised manuscript (Appendix G) to provide a more comprehensive understanding of VPG-C's capabilities.

(2.2) GPT-4 Evaluation on MinGPT-4, LLaVA, MIMIC-IT Datasets. We would like to respectfully clarify that the Mini-GPT4, LLaVA-Instruct-150K, and MIMIC-IT datasets are generated by LLMs and are primarily used for instruction tuning. These datasets do not have corresponding human-annotated test sets that are typically used for evaluation purposes. This unique nature of the datasets presents a challenge in terms of conventional evaluation protocols. Therefore, we leverage GPT-4 (text-only) to evaluate the quality of our model’s generated responses on these datasets.

Specifically, we randomly sample 200 examples from each of these three datasets. The sampled instruction-response pairs, along with additional object bounding boxes and captions for the images, are input into GPT-4 as references. Following recent studies [Liu et al., 2023; Yin et al, 2023], we leverage GPT-4 to evaluate the helpfulness, relevance, accuracy, and level of details of the responses from MLLMs. GPT-4 provides an overall score on a scale from 1 to 10 for each response, where a higher score indicates better performance.

As shown in the following table, we compare our VPG-C with InstructBLIP, MiniGPT-4, and LLaVA. While LLaVA and MiniGPT-4 have encountered some of the testing data during training, our VPG-C still achieves competitive performance. Particularly, our VPG-C demonstrates significant superiority on the challenging MIMIC-IT dataset, of which each instruction is accompanied by complex multi-modal in-context information.

(+ indicates that the testing samples have been used during the training stage.)

(2.3) GPT-4V Evaluation on MinGPT-4, LLaVA, MIMIC-IT Datasets. To further illustrate the effectiveness of our method, we use recently opened GPT-4V, which can directly take images as input. Since the API interface is not available and there are strict limits on the number of uses, we randomly sample 30 examples from each dataset and manually prompt GPT-4V via web interface. We use similar prompt templates as a recent study [Yin et al., 2023]. The following table verifies the consistent superiority of our VPG-C on these complicated instruction data.

(+ indicates that the testing samples have been used during the training stage.)

Thank you once again for your constructive suggestions, which have been instrumental in enhancing the quality of our research! We have now incorporated these experiments into our revised manuscript (Appendix H, I).

References

[Xu et al., 2023] LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. Arxiv 2306.09265.

[Liu et al., 2023] Improved Baselines with Visual Instruction Tuning. Arxiv 2310.03744.

[Yin et al., 2023] Woodpecker: Hallucination Correction for Multimodal Large Language Models. Arxiv 2310.16045.

Q3: In multimodal dialogues, does the model need to update guidance multiple times when providing multiple answers? In other words, for each new additional question, does the model need to run the whole model to obtaion all hidden states and can't use the existing qkv cache?

A3: Thank you for your insightful query. In our experiments, we update the guidance for each new question. Technically, it's feasible to reuse guidance generated from previous questions, allowing the model to use the existing qkv cache efficiently. This introduces minimal extra time overhead. However, each question in a dialogue may focus on different aspects of the image, making it sometimes necessary to reintegrate new residual details for a new question. Regenerating guidance for each question could theoretically improve response reliability but at the cost of increased time overhead.

In the following table, we present a comparison of the average test time and performance across each dataset in multimodal dialogue tasks within DEMON benchmark. We can see that:

• when regenerating guidance for each new additional question, the extra time overhead might not be significant (the test time is still less than that of the LLaVA model). Moreover, this approach does indeed achieve higher performance compared to other methods.

• even when reusing previously generated guidance for each new question, our method still outperforms other baselines in terms of performance, while maintaining a similar level of time expenditure.

Overall, the decision of whether to regenerate guidance for each new question in multi-modal dialogue is flexible and can be user-determined, based on the specific requirements and context of the scenario.

Thank you for your kind feedback and valuable comments. We have revised our manuscript and addressed several points you mentioned.

Q1: The paper does not mention the setting for instruction tuning. My understanding is that after using the synthetic discriminative training strategy, the model automatically acquires the ability to follow instructions without needing an instruction tuning phase.

A1: Thank you for your keen observation regarding our training approach. You are absolutely correct: the synthetic discriminative training strategy unleashes the model's instruction understanding ability, without further supervised instruction tuning stage.

Q2: The ablation in the paper validates the effectiveness of several proposed modules. But can VPG-C be applied to a finetuning setting, such as llava/minigpt4?

A2: Thank you for the constructive suggestion. As you nicely suggested, we adopt our VPG-C module to fine-tune LLaVA and MiniGPT-4 and report the results as follows.

(2.1) Evaluation on DEMON Benchmark. As shown in the following table, our method can be well adapted to LLaVA and MiniGPT-4 in a fine-tuning setting, empowering them to effectively attend to fine-grained details. For instance, on Visual Relation Inference, VPG-C brings about an improvement of 10.91 and 13.47 points for LLaVA and MiniGPT-4, respectively. Also, on Multimodal Dialogue, VPG-C largely improves LLaVA and MiniGPT-4 by 9.22 and 8.2 points, respectively. These results indicate that VPG-C enables them to effectively associate the interleaved images mentioned in the complicated context.

(2.2) Evaluation on MME Benchmark. Furthermore, we report the results on MME benchmark. The following table validates that our VPG-C is also beneficial to the perception and cognition abilities of LLaVA and MiniGPT-4.

Based on the above experiments, we believe that VPG-C is a reliable fine-tuning method to enhance the existing MLLMs' perception and reasoning abilities for visual inputs. Thank you once again for your valuable suggestions! We have now incorporated these experiments into our revised manuscript (Appendix F).

Thanks for the authors’ efforts in rebuttal. I keep my score as it is.

We sincerely thank you for your comprehensive comments and constructive advice. We will explain your concern as follows.

Q1: The method contains several steps and is thus quite complicated. It may be hard to reproduce the whole framework in different settings and code bases.

A1: We apologize if our manuscript gave the impression that the implementation of VPG-C is complex. Allow us to clarify some key points that we hope will address your concerns.

(1.1) Simplicity and Compatibility of VPG-C. Contrary to the perceived complexity, the implementation of VPG-C is quite straightforward. It does not require any modifications to the internal structures of either the VPG or the LLM. VPG-C performs like a prompt tuning approach at the intermediate layer of an LLM. Furthermore, as demonstrated in our experiments in Section 4.4 (specifically Table 4 Rows 7, 8, and 9), VPG-C is compatible with different VPGs and LLM backbones, and achieves performance improvements across these variations. This underscores the method's simplicity and adaptability across different models.

(1.2) Extended Experiments. To further support our claim, we have included additional experimental results in the following table. We apply our method to fine-tune LLaVA and MiniGPT-4, which are implemented with different code bases. The results show consistent performance improvements, indicating the adaptability and effectiveness of VPG-C across various settings and code bases.

(1.3) Future Improvements. We greatly appreciate your suggestion and will optimize our open-source project and documents. We recognize the importance of reproducibility and ease of use in the research community. In response, we are committed to enhancing our code repository with more detailed documentation and examples, making it more accessible for researchers and practitioners who wish to apply our method in different contexts.

We hope this explanation clarifies the simplicity and practicality of our method. We are grateful for the opportunity to improve our work based on your insightful comments and look forward to further discussions.

We sincerely thank you for the valuable comments. We are encouraged to see that our work is recognized as fairly motivated and encouraging. We will explain your concern as follows.

Q1: As the model is reusing its intermediate features via a feedback loop system, this will significantly increase the training and testing time and might not be batch friendly during inference. How does the throughput of this technique compares against previous methods?

A1: Thank you for raising an important concern. Our model derives the guidance from the intermediate layer of the LLM and seamlessly integrates the missing visual details into the subsequent layer via a skip connection. Therefore, our model does not introduce any additional LLM forward propagation processes and only requires an additional call of the VPG. Concerning time overhead, the forward propagation time of our 107M VPG is considerably less than that of a 7B LLM. As a result, the additional call to VPG does not significantly increase the overall training or testing time. Regarding memory usage, we effectively reuse the original VPG and the initial visual features generated by the VIT. VPG-C introduces minimal new model parameters (i.e., a set of query embeddings and two linear projection layers). Hence, the additional memory overhead is minimal.

Next, we will separately compare the throughput of VPG-C with MiniGPT4, InstructBLIP, and LLaVA during both the training and inference stages.

(1.1) Training Efficiency. Despite the additional VPG call, the overall impact on training time is mitigated by its inherently lightweight design. Since our proposed VPG-C module is lightweight with fewer trainable parameters (6.3M) compared to InstructBLIP (fine-tuning the Q-former) and LLaVA (fine-tuning the LLM), VPG-C requires less time overhead during the stages of backward propagation and parameter updating. To validate the above viewpoint, we use the same image-text pairs to train some MLLMs with a batch size of 48, ensuring identical hyperparameters. As shown in the following table, the throughput of VPG-C during training is slightly lower than MiniGPT4 but higher than InstructBLIP (and LLaVA).

(*Estimated, considering LLaVA model requires fine-tuning of the whole LLM, its training time per iteration is significantly higher than that of the other three models.)

(1.2) Inference Efficiency. Our tests using DEMON benchmark samples reveal that VPG-C only marginally extends the generation time (by approximately 6.3%). As shown in the following table, VPG-C's throughput during inference is competitive, only slightly lower than MiniGPT4 but higher than both InstructBLIP and LLaVA. Importantly, its GPU memory consumption is only marginally increased, ensuring batch efficiency.

In summary, while our model introduces a modest increase in processing time, it remains efficient and batch-friendly, particularly when considering its enhanced capabilities in completing visual details into MLLMs. We believe these trade-offs are justified given the model's performance gains.

Thank you once again for your constructive suggestion, which has been instrumental in enhancing the quality of our research! We have now incorporated the efficiency analysis into our revised manuscript (Appendix E).