MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

Thanks for the valuable and encouraging comments! Our point-by-point responses to the reviewer's mentioned concerns are provided as follows.

W 1: Evaluation Limitation

Response:

Thank you for your valuable suggestion. In order to draw comparisons with MIMIC-IT, we constructed a dataset of 3,000 evolutionary data points using MIMIC-IT and GPT4o-mini based on a seed set of 1,000 data points. We conducted comparative experiments with MMEvol, with the results presented in the table below. Under stringent conditions of fairness (seed data, evolutionary API, model architecture), MMEvol achieved an average lead of 6.4 points, demonstrating particularly significant superiority on the RealWorldQA task. This highlights the effectiveness of MMEvol.

W 2: Absence of Failure Case Study

Response:

Thank you very much for your detailed review. We have included the missing Failure Case Study in Figure 14 and highlighted it in red.

W 2: Technical Contribution

Response:

We further elucidate the differences between MMInstruct, MIMIC-IT, and MMEvol from the perspectives of complexity and diversity.

1. MMEvol is characterized by its elegance and simplicity, enabling the completion of an arbitrary number of task expansions using a unified prompt and straightforward image data. In contrast, MIMIC-IT and MMInstruct employ more complex pipelines and require a limited set of pre-defined tasks. For instance, MIMIC-IT necessitates both image and video data to accomplish six specific tasks. MMEvol transcends the limitations of pre-set tasks by generating new tasks, thus eliminating the need for intricate manual multi-task designs, making it more efficient and effective, with a higher degree of diversity.

2. MMEvol continuously iterates through comparison and evolution based on existing questions and answers via an instruction-driven evolution approach, generating more complex tasks. Conversely, MIMIC-IT and MMInstruct create new data by repeatedly inputting new images into pre-defined tasks through rewriting. The former has a clear objective of increasing complexity, while the latter merely reprocesses input data within a limited task framework. This results in MMEvol achieving superior data complexity and quality with only 480K evolution data, while MIMIC-IT requires 2.8M.

Q 1: why does negative scaling happen at step 6k-7k? Is it related to certain instruction augmentation techniques?

Response:

The issue of multi-task training conflicts exists within the framework of multimodal instruction fine-tuning [1]. The various capabilities of multimodal models do not enhance comprehensively without conflict. It is reasonable that negative scaling occurs between steps 6k and 7k. We did not employ any instruction enhancement techniques; rather, we simply present the results of our experiments.

[1] Chen Wei, et al."LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs" arXiv preprint arXiv:2401.16160 (2024).

Q 2: Table 1: What are the number of instructions for each row?

Response:

For each row, we employ 6K instructions for the ablation study. Here, we clarify that we utilized an equivalent amount of 6K data for each row.

Q 3: Table 2: What's Qwen2-7B baseline performance?

Response:

Thank you for your invaluable suggestions. We have added the Qwen2-7B baseline trained on the seed data (highlighted in red) to Table 2 of the paper.

Dear Reviewer u5c1:

Thank you for your extensive review and constructive comments on our manuscript. We have earnestly worked on resolving all issues highlighted and have provided comprehensive responses to your queries. As the review deadline is imminent, we humbly request your reevaluation of our revised submission. Please feel free to reach out if any further clarification is required from our side.

Your support is highly valued, and we thank you in advance for your consideration.

With gratitude,

Authors

Dear Reviewer u5c1,

I hope this message finds you well. We have carefully considered your feedback and have made significant improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations? We are eager to address any further queries you might have.

Thank you for your invaluable support and consideration.

Warm regards,

Authors

Dear Reviewer u5c1,

We genuinely value the time and effort you have dedicated to reviewing our manuscript. We have responded comprehensively to your comments and made the necessary adjustments. As the deadline for discussion nears, we kindly ask if you could review our updated paper. We are eager to address any additional questions that may arise.

Thank you very much for your continued support and assistance.

Sincerely,

Authors

Thanks for the valuable and encouraging comments! Our point-by-point responses to the reviewer's mentioned concerns are provided as follows.

W 1: Evolving multimodal dataset makes sense and is so interesting but actual performnace improvements are too marginal in my perspective: 2~3% because evaluating language ability can improve large margin if it is really contrbutional approach.

Response:

We emphasize our 2-3% performance enhancement from the following points:

1. The data we evolved is relatively limited. The 163K dataset constitutes only 15% of the total dataset (1.1M).

2. The rewriting model we employed, GPT-4o-mini, is cost-effective but not optimal. If we utilize a more advanced open-source model such as Qwen2VL with 72 billion parameters, we could achieve further improvements.

3. In contrast to Cambrian's dataset of 7 million, which includes an additional 6 million high-quality instruction data, we have achieved better results using only 480K data points (8%) evolved from a limited seed dataset.

Considering these three points, the fact that we achieved a 2-3% overall performance improvement with such a small amount of evolved data is quite acceptable. Should we scale up our efforts and use a more powerful MLLM, the results would be very promising.

W 2: Experiments should be compared for fair comparison where same architecture, same dataset to provide the effectiveness of MMEvol.

Response:

To conduct a fair comparison, we randomly downsampled the same number of data points from MMEvol for a rigorous comparative experiment. The results of the experiment are presented in the table below. Under the same data volume and model architecture, MMEvol-8B achieved an average improvement of approximately 2.7 points, demonstrating the effectiveness of our approach.

W 3: What kind of dataset samples are more effective to be applited by MMEvol like Math, code, or anyhting?

Response:

Thank you for your insightful concern. As we are solely focused on the textual portion of the instructional data during the evolutionary process, without supplementing new images, the text-centric multimodal instructional data type demonstrates greater efficiency and a higher success rate in terms of evolution. For instance, the success rate of evolutionary processes is significantly higher for types of data related to code and creative tasks. In contrast, for scientific diagrams (such as those in chemistry, physics, and mathematics), the success rate of evolution is relatively lower due to constraints imposed by the types and quantities of images available.

Thank you again for your insightful comments. If you have other comments, we are happy to address them to polish this work. We look forward to contributing to the development of both the Multi-Modal research and the open-source community.

Dear Reviewer h9M6:

We hope this message finds you well. We deeply appreciate your thoughtful feedback and the attention you’ve given to our manuscript. All concerns have been thoroughly addressed, and we wish to invite you to review the manuscript once more. With the deadline approaching, we would be grateful if you could confirm that all uncertainties have been resolved. We are ready to assist you with any further clarifications.

Thanks for your cooperation.

Kind regards,

Authors

Dear Reviewer h9M6,

We want to express our gratitude for your thorough review and helpful comments on our manuscript. We've diligently worked on incorporating your suggestions and believe the revised version is much stronger. The deadline for discussions is approaching, and we would appreciate your feedback on our revisions at your earliest convenience. Your thoughtful evaluation is crucial to us.

Thank you for your understanding and assistance.

Kind regards,

Authors

Dear Reviewer h9M6,

We are truly grateful for the thoughtful review you provided. We have taken all your feedback into consideration and revised the paper accordingly. Could you possibly re-evaluate our submission given the updates we have made? Your further feedback would be greatly appreciated, and we are prepared to clarify any remaining points of confusion.

Many thanks,

Authors

Dear Reviewer h9M6,

Your insights have been invaluable in refining our work, and we have diligently addressed each of your comments. As we approach the discussion deadline, we kindly ask if you could reassess our revised manuscript. We are more than willing to engage in further dialogue to ensure all your concerns are fully resolved.

Thank you for your attention to this matter.

Kind regards,

Authors

Thanks for the valuable and encouraging comments! Our point-by-point responses to the reviewer's mentioned concerns are provided as follows.

W 1: This leads to an unfair comparison, as in most cases, more data leads to better performance.

Response:

We sincerely appreciate the detailed review comments. To ensure a fair comparison, we present the following experimental results. We downsampled MMevol to 0.8M and conducted a comparative experiment using LLaVA-Next. As shown in the table below, MMevol achieved an overall improvement of 2.7, demonstrating the effectiveness of the method.

W 2: Is the model capable of finding out bad cases generated by itself? Further, I'm wondering about the fail rate and elimination rate of the proposed method.

Response:

To investigate the reliability of the rewrites produced by GPT-4-o-mini, we conducted a manual evaluation of the data before and after the evolution process. Specifically, we first extracted 30 images of various types from the seed data to ensure diversity, keeping 5 relevant question-answer pairs for each image. Subsequently, we carried out the corresponding evolution in three different directions, ultimately obtaining 450 evolved question-answer pairs, which were then subject to scoring and filtering. The results were distributed among five experts for manual evaluation of the accuracy of the model evolution and the scoring filter. The data is summarized in the table below. From the table, it is evident that the average success rate of evolution using MLLM can reach 90%, while the accuracy of the scoring filter can achieve 94%, indicating the reliability of MMEovel. Additionally, we provide detailed scoring cases in Figure 15, highlighted in red.

Dear Reviewer pN7Z:

We are truly grateful for your insightful comments and the guidance you provided during your review of our paper. We are pleased to inform you that we have addressed all points raised and have made significant improvements. As the discussion phase draws near, we kindly request your reevaluation at your earliest convenience. Should any questions remain, we are at your disposal to clarify them promptly.

Thank you for your time and understanding.

Sincerely,

Authors

Dear Reviewer pN7Z,

Firstly, thank you for your meticulous review and constructive feedback. We have addressed all the points you raised and have clarified any ambiguities in the revised manuscript. As the deadline for discussions nears, we kindly ask if you could review our changes and provide any further comments. Your inputs are invaluable, and we're committed to ensuring the manuscript meets your standards.

Thank you for your continued support.

Best wishes,

Authors

Dear Reviewer pN7Z,

Your insights have been invaluable in refining our work, and we have diligently addressed each of your comments. As we approach the discussion deadline, we kindly ask if you could reassess our revised manuscript. We are more than willing to engage in further dialogue to ensure all your concerns are fully resolved.

Thank you for your attention to this matter.

Kind regards,

Authors

Thanks for the valuable and encouraging comments! Our point-by-point responses to the reviewer's mentioned concerns are provided as follows.

W 1The entire data improvement framework relies on closed-source frontier models like GPT-4, which suggests a form of knowledge distillation from these models but may limit the ability to scale to larger datasets. Additionally, the strong dependence on models like GPT-4 reduces the framework's interpretability.

Response:

In this study, we employ the GPT-4o mini model as a cost-effective alternative to the GPT-4 for data evolution. Our results demonstrate that its performance is comparable to that of the latter one, while offering lower costs (15K$vs. 600$). Additionally, the GPT-4o mini's performance aligns closely with established high-performance open-source alternatives, making it a favorable choice for our evolution. We present the results of 3K data evolved using the open-source model Qwen2VL 72B, as illustrated in the table below.

Compared to GPT-4o-mini, utilizing the more powerful open-source Qwen2VL 72B yields superior results, demonstrating the scalability and practicality of our approach.

Q1&W 2: The authors do not provide an analysis of the reliability of using GPT-4 as a data rewriter. How reliable is GPT-4 as a data rewriter? How can the quality of the rewritten data be evaluated?

Response:

To investigate the reliability of the rewrites produced by GPT-4-o-mini, we conducted a manual evaluation of the data before and after the evolution process. Specifically, we first extracted 30 images of various types from the seed data to ensure diversity, keeping 5 relevant question-answer pairs for each image. Subsequently, we carried out the corresponding evolution in three different directions, ultimately obtaining 450 evolved question-answer pairs, which were then subject to scoring and filtering. The results were distributed among five experts for manual evaluation of the accuracy of the model evolution and the scoring filter. The data is summarized in the table below. From the table, it is evident that the average success rate of evolution using MLLM can reach 90%, while the accuracy of the scoring filter can achieve 94%, indicating the reliability of MMEovel. Additionally, we provide detailed scoring cases in Figure 15, highlighted in red.

Dear Reviewer 8Cfc,

I hope this email finds you in good spirits. We are grateful for your detailed evaluation and have worked hard to address your concerns. With the discussion deadline fast approaching, we would appreciate it if you could revisit the manuscript to see if the adjustments align with your expectations. Your expertise continues to play a pivotal role in refining our work.

Thank you once again for your dedication.

Sincerely,

Authors

Dear Reviewer 8Cfc,

Thank you very much for your detailed review and constructive feedback. We have carefully revised the manuscript to resolve the issues you've pointed out. With the discussion deadline approaching, we would be grateful if you could review our changes. Should you require any further clarifications, please let us know, and we will gladly provide them promptly.

Best wishes,

Authors

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W 1: First, the prompt is clear in Fig 4 - Fig 7, can you explictly describe what model, commercial API or other method is used to generate the evolutoin instruction data from the prompt templates?

Response:

We utilized GPT-4o-mini, and as mentioned in Appendix C and the Limitation Section, we will include a clearer description in the revised version of the main paper.

W 2: It is mentioned pseudo function calling is used for visual reasoning evolution, can you describe the setup of function call, the model or template used to generate the function call, and how it is integrataed to the visual reason evolution process?

Response:

We have provided the following example regarding Figure 3 of the paper and offered a further explanation of the evolution of cognitive reasoning.

Seed Sample:

{
  "objects": ["window", "couch", "vase", ...],
  "skills": ["Grounding Ability", "Context Understanding Ability", ...],
  "format": "Normal",
  "question": "Which is closer to the window, the couch or the vase?",
  "steps": [],
  "answer": "The couch."
}

Evolved Sample:

{
  "objects": ["window", "couch", "vase", ...],
  "skills": ["Grounding Ability", "Context Understanding Ability", ...],
  "format": "Complex reasoning",
  "question": "Which is closer to the window, the couch or the vase?",
  "steps": [
    {
      "manipulation": "grounding_1(window) -> bbx_1",
      "description": "Locate the window in the room and return its bounding box as bbx_1."
    },
    {
      "manipulation": "grounding_2(couch) -> bbx_2",
      "description": "Locate the couch in the room and return its bounding box as bbx_2."
    },
    {
      "manipulation": "grounding_3(vase) -> bbx_3",
      "description": "Locate the vase in the room and return its bounding box as bbx_3."
    },
    {
      "manipulation": "Calculate([window, couch/vase]) -> res_i",
      "description": "Calculate the spatial distance between the window and the couch/vase."
    }
  ],
  "answer": "First, based on the detected bounding boxes of the window [0.2, 0.23, 0.57, 0.4] and the couch [0.17, 0.43, 0.83, 0.79], it can be inferred that the couch is positioned below the window. Further, according to the location of the vase [0.5, 0.51, 0.58, 0.72], it is located near the center of the room. Therefore, the couch is closer to the window."
}

In this example, we instruct the model to generate a series of reasoning steps (the "steps" field) using pseudo function calls. These steps simulate operations such as grounding objects and calculating spatial relationships. By incorporating these steps into the evolution process, we enhance the model's ability to perform complex visual reasoning. It is important to note that we achieve both processes simultaneously by incorporating requirements in the prompt, which ultimately yields more complex instructions and enhances the model's reasoning capabilities.

W 3: Third, it is mentioned MLLM is used for rewriting, can you describe details on what MLLM is used, how it is used (prompt used), evalution or abalation of the significance of this rewriting step?

Response:

Kindly refer to W1&W2 and the corresponding replies. We utilized the GPT-4o-mini model and formatted the instruction data based directly on the prompts, which were then fed into the model to generate and parse the evolved data. To investigate the impact of removing the chain of reasoning on inductive rewriting, we also conducted the following ablation experiments on 3K evolved data.

As we can see from the table, the induction of reasoning chains and rewriting requirements into the answer are crucial.

Dear Reviewer 1KdT,

Thank you for contributing your time and expertise to review our manuscript. We've taken your insightful comments seriously and amended the paper accordingly. As we approach the discussion deadline, we are eager to hear your thoughts on the revised version. Should any points still need clarification, we're ready to assist promptly.

Thank you for your indispensable guidance.

With gratitude,

Authors

Dear Reviewer 1KdT,

We deeply appreciate the time and effort you have invested in reviewing our paper. We have thoroughly addressed your valuable comments and made the necessary revisions. Could you kindly re-evaluate our manuscript at your earliest convenience? We are more than willing to discuss any remaining concerns you might have.

Thank you for your understanding and cooperation.

Best regards,

Authors

Dear Reviewer 1KdT,

We hope this message finds you well.

We have carefully addressed your questions and concerns in the rebuttal, including conducting additional experiments and providing detailed clarifications.

As the rebuttal deadline is approaching, we kindly invite you to join the discussion. We would greatly appreciate it if you could reconsider your rating, provided all your concerns have been addressed. If you have any additional questions, please do not hesitate to let us know. We are more than happy to provide further clarifications.

Thank you again for your careful review and valuable suggestions!

Best,

Authors

Dear Reviewer 1KdT,

Thank you for taking the time to review our responses and for updating your rating! We sincerely appreciate your recognition of our efforts! We strongly agree that more exploration of synthetic data for VLMs is needed. The lack of synthetic data, especially data with diversity and complexity, seriously hinders the further improvement of VLM performance. Besides, we will add the ablations to the final version of this paper.

Thank you once again for your valuable suggestions and constructive feedback throughout the entire review process!

Best,

Authors