2. About comparisons of SFT results on various architectures with other similar datasets

To further validate the effectiveness of TaskGalaxy and to supplement the performance comparison with similar datasets, we conducted second-stage fine-tuning experiments using four datasets: TaskGalaxy, LLaVA-665K, LLaVA-OneVision, and ShareGPT4v. These datasets were used to fine-tune two models: LLaVA-v1.5-7b and InternVL-chat-v1.0-7b. To ensure fairness, we randomly sampled an equal number of data samples from the latter three datasets to match the size of TaskGalaxy. This approach highlights the impact of TaskGalaxy's rich and diverse task types. The experimental results are summarized in the following table:

LLaVA-v1.5-7b:

InternVL-Chat-V1.0-7b:

We sincerely appreciate your valuable comments. We have conducted more experiments to thoroughly address your concerns, and your suggestions and questions have been highly valuable in enhancing our work. Welcome any additional discussions or feedback.

Dear Reviewer EFVj：

Thank you for your constructive feedback and greatly appreciate your positive comments. We apologize for not providing reviewers with a link to TaskGalaxy. The link to the dataset is provided below: TaskGalaxy Link

1.About advantages of the proposed dataset

To explore the advantages proposed for our dataset, first of all, we would like to explain the motivation for generating this dataset: because the data requirements for fine-tuning the current large multimodal models are the quality, quantity, and diversity of the data, and we are more concerned about the diversity and quality of the data, which directly determines the generalization capability of the multimodal models, and the diversity of the data is reflected in the diversity of the tasks, and we start from the perspective of the task diversity to construct a high-quality multimodal instruction fine-tuning dataset that covers as many task types as possible in natural world scenarios.

In order to explore the diversity of tasks, there are some previous works, but the generation of task types is very time-consuming and labor-intensive, and requires experts to manually generate and annotate, etc., and the generation of task types is limited to about 200 at most, which is difficult to cover the variety of task types in the open world. To address the above problems on the one hand, in order to liberate manpower as much as possible and generate as many task types as possible in the open world, the pipeline of our data generation is to utilize a few hierarchical task types generated by human beings as the initial seed, and then build a detailed prompt to let the robust closed-source model GPT-4o generate the task types hierarchically in a cyclic manner, which liberates manpower, and the task types are also limited by the prompt, and the task types are also limited by the prompt. The task types are also restricted to not overlap as much as possible by the prompt, so that the quantity and quality of the generated hierarchical task types are guaranteed.

In addition, we designed a two-stage task type and open source image matching strategy, as well as a joint filtering strategy of multiple open source models on the question-answer pairs related to the generated task types to further ensure the quality of the data. Our generation pipeline takes into account both task diversity and data quality. Moreover, by utilizing our generation strategy, it is theoretically possible to flexibly expand the task types and question-answer samples. Our data generation strategy and the generated dataset can provide a reference for the instruction fine-tuning dataset community.

3. About the CLIP model

In the stage of match and filter, CLIP-L/14 published by openAI is adopted. This model uses a ViT-L/14 Trasnformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. So we utilize it for the initial match and filter of every image and task type pools.

4. About the acc change on various architecture

In Table 2 of the article, we compared the models using two different architectures for each model across two architecture sizes. After incorporating the fine-tuning of our proposed dataset and excluding MME and POPE, which are not applicable to these two rubrics, the average accuracy for the remaining benchmarks is as follows: for llava-v1.5-7b and llava-v1.5-13b, the scores are 46.26 and 46.39, respectively. For specific benchmarks, the results are: MMB's 68.04 and 69.85, LlaVAw's 56.3 and 63.1, MMVet's 29.9 and 34.4, HalluBench's 51.74 and 53.21, and SEED's 60.28 and 61.22. A similar trend is observed for internevl-chat-v1.0-7b and -13b, indicating that a larger number of parameters in model training corresponds to higher accuracy, which aligns with our intuition.

About details of ethics concerns:

1. The following additions have been made to the licensing information for image sources in Table 1 and will be incorporated in subsequent releases: | Dataset | License | |----------------|-------------------| | ALLaVA | Apache License 2.0| | VisualGenome | CC BY 4.0 | | MathV360K | Apache License 2.0| | ShareGPT4V | CC BY-NC 4.0 |

2. The license designated for the subsequent open-sourcing of the proposed dataset is CC BY-NC 4.0.

2. About baseline + TaskGalaxy data:

The performance improvement observed when combining the original instruction fine-tuning dataset with our TaskGalaxy dataset can be attributed to two key variables: data quantity and task diversity. Using the baseline as a benchmark, we analyze these two variables in the ablation experiments presented in Section 3.3 of the main text ([Fig. 6]). To evaluate the impact of data quantity, we fixed the total number of tasks and varied the maximum number of samples per task type. This adjustment allowed us to observe the effect of data quantity on the model's average accuracy, as shown on the left side of Fig. 6. The results demonstrate a clear trend: increasing data quantity improves performance. However, as our primary focus is on task diversity, we further explore its effect in the second analysis.

For task diversity, we fixed the total data quantity of TaskGalaxy and randomly sampled varying numbers of task types. The results, depicted on the right side of Fig. 6, reveal that including more diverse task types significantly enhances model performance for a fixed total data amount. This finding aligns with our expectations. Similar trends are observed in MME performance results in [Fig. 7].

To further demonstrate the impact of TaskGalaxy's task diversity on model performance, we compare the baseline fine-tuning data with TaskGalaxy fine-tuning data. The comparison results, summarized in the following table, clearly show that TaskGalaxy's task diversity compensates for the limitations of existing baseline fine-tuning datasets, improving both model performance and stability.

3. About Match and Filter by CLIP:

Our motivation for employing the CLIP model stems from its proficiency in graphic matching, which enables us to conduct an initial screening to identify a broad range of relevant task candidate sequences—up to 30 for each image. In our experiments, we observed that the candidate task lists generally exhibited a good match with the images. However, as reviewers have noted, there is a potential for some abstract concepts not aligning perfectly with the images, or that CLIP might not be the optimal tool for this purpose. Consequently, we implemented a secondary screening phase. This involves using the existing images and their corresponding task candidate sequences to prompt GPT-4o in designing a screening mechanism, instructing the model to ascertain which task types align more closely with the image content and are more suitable for generating relevant Q&A pairs. This secondary filtering process often results in highly compatible task types; an illustrative example is provided in Figure 1 in the Appendix.

The rationale behind utilizing CLIP is to efficiently filter certain task types and subsequently reduce the input length required for GPT-4o. Since it is impractical to present all task types to GPT-4o for direct screening, this approach helps streamline the process.

We appreciate your valuable comments. In response, we have provided a more detailed explanation of the task type generation process and validated the effectiveness of TaskGalaxy by separating it from the Baseline data and testing it with a stronger open-source model. If you have any further questions or suggestions, please feel free to reach out!

Dear Reviewer EeP4，

Thank you for your insightful feedback! Below, we address your concerns and questions individually:

1. About tasks and data

The issue of overlap among task types presents a significant concern. We generate task types for both primary and subordinate categories using prompts specifically designed to avoid overlap, thereby allowing GPT-4o to extend or create distinct task types. For instance, a prompt might be, “Please expand and supplement with new primary-level categories that do not belong to the aforementioned categories” (refer to Table 1 in the Appendix for additional prompts that yield non-overlapping task types). Ultimately, the quality of the generated task types hinges on GPT-4o. Our objective is to produce as many distinct task types as possible without human intervention.

To further illustrate the variance in similarity among the generated task types, we computed a similarity matrix for all hierarchical task types using CLIP's text encoder. We then identified the maximum similarity value between each sample and all other samples, which ranged from 29.53% to 71.85%. This indicates a relatively low level of similarity. In future work, task types exhibiting slightly higher similarity can be filtered for further optimization.

The diversity of tasks and data largely depends on the hierarchical generation capabilities of GPT-4o, as well as the inclusion of an extensive range of public image data sources. The tasks addressed and a select number of data samples are detailed in Appendices A.3 and A.4, encompassing a broad spectrum of tasks within the general domain.

To assess the characteristics of our dataset in relation to previous studies, we provide a comparative analysis between our dataset and previous work in the following tables:

4. About more advanced models:

For more advanced models, since LLaVA 1.6 has not officially released its second-stage instruction fine-tuning dataset, we utilize the InternVL-Chat-V2.0-8B model, which has made its second-stage instruction fine-tuning data publicly available. For the comparison, we randomly sample the same number of samples as TaskGalaxy from the officially disclosed instruction fine-tuning dataset.

We fine-tune InternVL-Chat-V2.0-8B using both the original instruction fine-tuning dataset and the TaskGalaxy instruction fine-tuning dataset, ensuring that the number of samples for each is consistent with TaskGalaxy. The performance comparison table between the original instruction fine-tuning dataset and the TaskGalaxy instruction fine-tuning dataset on InternVL-Chat-V2.0-8B is as follows:

As shown in the table above, after fine-tuning the model using the TaskGalaxy dataset and the original InternVL-Chat-V2.0-8B fine-tuned dataset, TaskGalaxy outperforms the original baseline dataset on 14 out of 16 benchmarks. For the remaining two benchmarks, ChartQA and LLaVA-in-the-wild, the performance difference compared to the baseline dataset is minimal (less than 0.5). This demonstrates that TaskGalaxy's enhancement of task diversity is also effective for more advanced models, providing a significant boost in overall performance.

About typo: We will correct the word 'illustration' in Figure 1 in a subsequent version.

Thank you for taking time to review our rebuttal.

About diversity：

While we cannot theoretically prove that GPT-4o can always increase the task diversity based on our proposed framework, we have empirically proved that GPT-4o does help increase the task diversity. If GPT-4o did not help increase the task diversity, adding more number of task types into the TaskGalaxy would not increase the task diversity of TaskGalaxy. In this case, TaskGalaxy with more task types would not result in better performance on several downstream benchmarks. In other words, two TaskGalaxys which contain different number of task types but same number of total data will result in similar performance on multiple benchmarks. However, as Figure 6[Left] in our manuscript shows, as the number of task types generated by our pipeline increases, the performance also increases on multiple benchmarks. These results strongly support that GPT-4o dose help increase the task diversity.

On the other hand, while we cannot theoretically prove that GPT-4o can reliably provide task diversity, the generated TaskGalaxy in this study did contain more diverse tasks (empirically proved as discussed above) and can be used for various downstream tasks in the SFT dataset research community. Our approach aims to minimize human intervention for generating a wide range of diverse and non-overlapping tasks. We have incorporated additional mechanisms to refine and validate task diversity. For instance, we utilize initial task seed provision, hierarchical prompt engineering, and task embedding analysis to differentiate between tasks. The ideal method would involve incorporating expert reviews to manually verify and refine task types. This would ensure a higher level of diversity, but it conflicts with our original goal of minimizing human involvement in the task generation process.

Despite its limitations, on diversity ensurance we believe this approach offers a valuable perspective on achieving reasonable task diversity under constrained human intervention.

About your concern about similarity scores which cannot fully substantiate claims of task diversity, we believe that similarity can still serve as an assessment indicator to evaluate the degree of overlap among tasks, reflecting the degree of diversity TO SOME EXTENT.

About more datasets:

To further highlight the advantages of TaskGalaxy, we compared its performance with that of several other SFT datasets, including LLaVA-OneVision[1], ALLaVA-4V[2], and Cambrian-1[3] in the following table. For a fair comparison, we randomly sampled the same number of samples from each dataset as in TaskGalaxy for fine-tuning. The experimental results demonstrate that TaskGalaxy consistently achieves the highest performance on most of he benchmarks across multiple model architectures, validating its effectiveness.

LLaVA-V1.5-7b:

InternVL-Chat-V1.0-7b:

Thank you again for your thoughtful comments. We sincerely appreciate the opportunity to engage in this meaningful discussion and welcome any further suggestions or feedbacks.

Reference [1] Li, Bo, et al."Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024).

[2] Chen, Guiming Hardy, et al. "Allava: Harnessing gpt4v-synthesized data for a lite vision-language model." arXiv preprint arXiv:2402.11684 (2024).

[3]Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." arXiv preprint arXiv:2406.16860 (2024).

Dear Reviewer VWt8,

We sincerely appreciate your positive feedback and constructive suggestions regarding our work. We understand your interest in the presentation and analysis of incorrect and illusory samples generated by the MLLM model during the dataset creation process. Additionally, you highlighted the importance of showcasing the samples that were discarded through our screening and matching strategy at various stages. We agree that including examples of samples screened out for containing illusory or incorrect responses would greatly enhance the reader's understanding of our dataset generation process.

In the field of multimodal large model fine-tuning, both data quality and diversity are crucial factors at this stage. A high-quality, diverse dataset with rich command fine-tuning capabilities is essential, and our work is centered around this focus. As you mentioned, the rejected samples in the second point partially correspond to those in the first point due to the illusions and erroneous responses generated by the MLLM during text generation. Here, we illustrate and analyze examples of samples that were filtered out throughout our entire data generation pipeline. For a detailed example, please refer to the PDF available at the anonymous link provided below:

---

Sample Analysis PDF

---

Thank you very much for your suggestion. Analyzing the samples filtered out in the data generation pipeline provides valuable insights into the quality of TaskGalaxy and helps to better understand the current challenges faced by MLLMs in data generation.

Dear Reviewer U9mU，

Thank you for your insightful feedback! Below, we address each of your concerns and questions in detail：

1. About task types:

The names of the task types for level 2/3 stratification and the corresponding data examples are partially presented in Appendices A.3 and A.4. The detailed statistics suggested by the reviewer are helpful for a deeper understanding of TaskGalaxy. As mentioned in the text, we aim to keep the sample sizes for each task as balanced as possible. We control the number of samples per task to be between 1 and 55. The table below lists tasks with the maximum sample size of 55 and the minimum sample size of 1.

For the issue of image resolution, we collect the image portion of each open source dataset, keeping their original image resolution, and for all models, the input image resolution is configured to 336 by 336 pixels.

For the distribution of task types across layers, the ratio of task types in the first three layers is 1:2:3, corresponding to 115, 2796, and 14,370, respectively.

2. Overall review of images sources

Considering the accessibility of data sources and the task-related nature of the image data we aim to mine, we have opted for open-source image data. The approximate data sources and their corresponding sample sizes are presented in Table 1 of the main text. To provide further insight into the image data, the following table presents the statistics of the sample sizes for the different data sources collected.

To further expand the existing image data, our next step is to use a generative model to create corresponding images for tasks that lack image matching.

5. About a detailed dataset card

To provide more statistics and usage information for TaskGalaxy, we have created a dataset card for TaskGalaxy following the suggestion. The dataset card is anonymously linked at the following address:

---

TaskGalaxy Data Card

---

3&4. About more convincing experiments:

To effectively demonstrate the validity of our dataset, we use LLaVA-665K, TaskGalaxy, LLaVA-OneVision, and ShareGPT4V individually during the fine-tuning phase. To control for variables, we randomly sample data from the other three datasets to match the size of the TaskGalaxy for fine-tuning LLaVA-v1.5-7b and InternVL-chat-V1.0. The results are presented in the table below.

LLaVA-v1.5-7b:

InternVL-Chat-V1.0-7b:

For the LLaVA-v1.5-7b model, TaskGalaxy demonstrates the strongest performance on 14 out of 16 benchmarks compared to the other three instruction fine-tuning datasets. Additionally, it achieves the highest average performance across all benchmarks, excluding MME. Similar trends are observed for the InternVL-Chat-v1.0-7b model, further reinforcing these findings. The experimental results clearly demonstrate that the diversity of task types in the TaskGalaxy data generation pipeline significantly enhances the model's generalization capability.

And we will add more works like ShareGPT4V in related work.

We would like to thank you for your insightful questions and suggestions regarding our work. The statistical analysis of task types, data sources, and other aspects during the data generation process has been instrumental in providing a more comprehensive understanding of TaskGalaxy. Additionally, the data cards offer a clear and intuitive way to visualize TaskGalaxy. We have also conducted further evaluations to validate the effectiveness of TaskGalaxy, and we warmly welcome more discussions on this topic!

Thanks for your reply and additional information.

The most important concern of mine is about the improvement of TaskGalaxy and relative datasets with the same quantity. As shown by experiments provided by the authors, results show that TaskGalaxy can bring more improvements than counterparts. I think this can resolve my concern about effectiveness.

However, scale and data sources of TaskGalaxy is considered to be not satisfying. With similar data amount, performance of models finetuned by llava-onevision is pretty bad, while models finetuned with original LLaVA-OneVision is much better. It shows that dataset scale is also important in MLLM finetuning. But the construction procedure of TaskGalaxy seems to be not scalable enough. For example, there are tasks having only 1 samples, indicating data samples related are hard to collect. I think TaskGalaxy can enhance diversity and quality of data sources further.

Overall, I would like to improve my rating slightly as some of my concerns have been resolved.