Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage

Thanks for your insightful and thorough review. We will address your concerns below.

W1. In Section 3.3, this work states, “For other needed files, we prompt GPT-4o mini to extend the file content and generate Python code to produce the files.” However, the methodology for file generation remains unclear. For example, if a file is required to be an MP4 video of Taylor Swift’s latest album, it’s uncertain how this content could be generated through Python code alone. Furthermore, if GPT-4o mini generates the Python code to produce such files, it raises concerns about data quality and how the model ensures that the generated content is not hallucinated.

R: In this work, we do not generate video files (e.g., MP4, MOV), and the current T3-Agent is not designed to handle video-based tasks. The primary reason is the lack of sufficiently reliable video tools. While we have explored video tasks, we found that even the most advanced tools (e.g., Gemini 1.5 Pro) struggle with challenges requiring fine-grained video understanding, complex video reasoning over long video sequences, and deep semantic interpretation.

In future work, we will address the challenges by exploring more advanced video tools and integrating them into the T3-Agent. We also acknowledge that hallucination remains a significant challenge. To address this, rather than using GPT-4o to generate Python codes for video creation, we will adopt a retrieval-based strategy (similar to the generation strategy of image tasks) to mitigate such risks in video tasks.

W2. While including a human study to assess dataset quality is commendable, having only five experienced raters for a subset of the data may be too limited, potentially introducing biases based on individual preferences. Gathering feedback from a larger pool of participants, even with fewer data points per person, could strengthen claims about the dataset's effectiveness. Additionally, comparing MM-Traj to filtered-out data may not yield meaningful insight. Instead, comparisons with other established tool-usage datasets would likely provide more meaningful insights.

R: Thanks for your comments. We add a more comprehensive user study during rebuttal, where we recruit 30 persons with rich programming and AI experience to evaluate the generated tasks and corresponding trajectories. Each person is tasked with assessing 20 tasks with trajectories, which are randomly selected and mixed from both MM-Traj and filtered-out data. We ask each one to provide scores (1-10) for the task quality and trajectory quality, and they do not know whether the data is from MM-Traj or filtered-out data. A low score means the quality is bad and a high score means its quality is good. The average scores are shown in Table J. The results clearly show that the used verifiers remove low-quality data. Due to the time limitation, we will further expand the scale of the user study in a revised version.

Table J: User study on the MM-Traj dataset

Thank you for your suggestion to compare MM-Traj with other established tool-usage datasets. However, it is difficult to make direct comparisons, due to the domain, complexity, and evaluation differences among these datasets. In the future, we will bridge this gap by introducing a unified inference formulation among these datasets and further conducting fair comparisons.

Thanks for your insightful and thorough review. We will address your concerns below.

W1. The T3-Agent exhibits a gap in programming capabilities, which leads to lower accuracy in predicted answers compared to GPT-4o.

R: The performance gap in programming can likely be attributed to the differences in model size and training data of the agent controller. Compared with the MiniCPM-V model in the T3-Agent, GPT-4o benefits from a larger model size and richer training corpus.

W2. While the paper acknowledges the T3-Agent’s limited programming capabilities, it does not suggest potential improvements or outline future directions to strengthen this aspect.

R: Thanks for your comment. In the future, we could improve the programming capabilities of the agent via two manners. (1) Use VLMs that are pre-trained for coding as the controller. (2) Combine code data (such as the PyTraceBugs dataset [d]) with our trajectory data in fine-tuning.

[d] PyTraceBugs: A Large Python Code Dataset for Supervised Machine Learning in Software Defect Prediction

W3. The reliance on GPT-4o mini throughout the pipeline raises questions about biases and limitations from this closed-source model. Exploring alternative methods or open-source models could enhance transparency and address these limitations.

R: We chose GPT-4o mini due to its well recognition within the research community. However, we emphasize that our method is not tied to specific models and can flexibly transfer to open-source alternatives. To demonstrate this point, we add an experiment where the GPT-4o mini model in the data synthesis pipeline is replaced by the open-source Qwen2.5-72B model. Using this setup, we collect 12K new data and fine-tune the MiniCPM-V-8.5B model. The resulting performance is evaluated on the GAIA dataset, with the results presented in Table I. These results demonstrate that the 12K data produces about 6% improvements over the untuned baseline, highlighting the model-agnostic nature of our approach. We will include these findings in the revised manuscript to emphasize the flexibility and compatibility of our method with open-source models.

Table I: Agent on the GAIA benchmark.

W4. What is $p_i$ in Equation 2?

R: Thanks for pointing it out. $p_i$ should be corrected as $t_i$, denoting the generated thought in the $i$-th step, and the Equation 2 is

$

\min \mathbb{E}_{(F^{\star}, Q, T, C, O, A)\sim \mathbb{D} } [ -\sum^{n}_i P(t_i , c_i| F^{\star}, Q, h_i)],

$

where we train the agent controller to fit the thought $t_i$ and code $c_i$. The equation will be revised in a new version.

Q1. I lean towards acceptance because the paper introduces a promising approach to enhancing the reasoning capabilities of multi-modal agents. The proposed data synthesis pipeline and the resulting MM-Traj dataset are significant contributions to the field, potentially advancing the state of multi-modal learning. However, the paper lacks a discussion on the T3-Agent's robust programming capabilities and does not explore how these might be improved. I would like the authors to comment on this aspect.

R: Thanks for your positive comment. To improve the programming capabilities of agents, there are two manners: (1) using VLMs pre-trained for coding and (2) combining code data with our trajectory data in fine-tuning. We will add the above discussion in the revised manuscript.

Thanks for your insightful and thorough review. We will address your concerns below.

W1. Interpretability: While the approach demonstrates performance gains, the interpretability of results remains limited. Additional analyses, such as ablation studies or attention maps, would be beneficial to understand how each modality contributes to the decision-making process. For example paper first generates queries first without files before relevant queries, what is the impact if we don't do that, or this is based on some past work/observations?

R: Thanks for your comments. For the interpretability of our method, we add ablation experiments to show the contributions of different modalities in decision-making. Ablation results on the GTA dataset are shown in Table E, where removing the image modality reduces the performance by 40%, highlighting the importance of input images.

Table E: Ablation on the GTA Benchmark

Regarding your concern "first generate queries without files before relevant queries", we adopted this strategy based on our observations, as it leads to more natural synthetic tasks. For complex tasks involving multiple files, first generating files and then queries often result in less natural tasks, as the generated files within one query may have limited or unrelated connections. By generating queries first and then creating files accordingly, we can ensure stronger coherence and relevance among the files based on the information in queries.

W2. Scalability: The paper does not thoroughly address the scalability of the proposed method, particularly as the number of modalities or the dataset size increases. It would be beneficial to test how the method's performance and computational requirements scale with additional modalities or larger datasets. For example, experiments that measure latency, memory usage, and accuracy as more data is introduced could illustrate the framework's robustness and its viability in resource-constrained or high-throughput environments.

R: For the scalability of modalities, our method is able to extend to additional modalities by incorporating more tools and leveraging advanced multimodal models. For example, to extend our method to the video modality, we can integrate a video search model into the data synthesis pipeline and replace the MiniCPM-V model with a video-language model for the agent. This approach ensures seamless adaptation to new modalities while maintaining efficiency and coherence.

For the scalability of dataset size, we add experiments in Table F to show the agent's performance on the GTA benchmark as the dataset size increases. With the increase of data number, the agent achieves better performance, the memory consumption is constant, and the time consumption linear increases. Compared with the accuracy improvements, we think that the consumption of memory and time is acceptable.

Table F: Ablation on the GTA Benchmark

We agree with the reviewer about the importance of the scalability of modalities and dataset size. However, our current work mainly focuses on demonstrating the effectiveness of the multimodal agent tuning method. We will expand to additional modalities and explore engineering optimizations in the future, to enhance the applicability of our method by using fewer resources but for more tasks.

W3. User Study: To evaluate the practical usability of the framework, a small user study or qualitative feedback from users would provide valuable insights into the query handling experience. Specifically, gathering feedback on aspects like ease of use, perceived accuracy, responsiveness to complex queries, and the intuitiveness of the tool-usage process could highlight areas for refinement in real-world settings.

R: Thanks for your comments. We add a user study about agent outputs on the GTA benchmark. We recruited 20 participants, each evaluating 20 tasks with agent outputs, where the agent is with or without fine-tuning. The outputs of the agent (w/ or w/o tuning) were shuffled for each task, and the participants were not informed about the source, ensuring an unbiased assessment. The participants were asked to provide the preference of the two agent outputs, based on the accuracy, helpfulness, and relevance. We measured the percentages of results, as shown in Table G. Outputs from the tuned agent has a significantly high preference, indicating its better performance in solving practical tasks.

Table G: User study for agent outputs on the GTA benchmark.

Thanks for your insightful and thorough review. We will address your concerns one by one.

W1. Verifying the output of an LLM by the LLM itself does not seem accurate. I am skeptical about the quality of the generated MM-Traj dataset.

R: Using LLMs to verify the quality of LLM outputs has shown effectiveness in multiple works, such as [a] verifying the generated instructions and [b] verifying the generated plans. Inspired by them, we argue that using LLMs can also verify the synthetic tasks and trajectories. The ablation experiments about the used verifiers are shown in Table 5, where the verifiers lead to about 2% improvements on both GTA and GAIA benchmarks. Meanwhile, a more solid user study (involving more people and more data points) is presented in the response to W2, which also confirms that the verifiers can filter our low-quality data.

To evaluate the quality of the remaining data in the MM-Traj dataset, we have compared models with and without using the MM-Traj dataset, as shown in Table 2 and Table 3 of the manuscript. Using our dataset leads to 18.59% and 7.88% improvements on the GTA and GAIA benchmarks, respectively, demonstrating the effectiveness of the MM-Traj dataset.

[a] Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023.

[b] APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. NeurIPS 2024.

W2. You need quantitative verification for the dataset. (It is not clear whether a user study involving a few people on 100 data points out of 15K would provide sufficient confidence.)

R: To address your concern, we conduct a more extensive and robust user study during rebuttal. Specifically, we recruited 30 participants with rich experience in programming and AI to evaluate the quality of the generated tasks and trajectories. Each participant was tasked with assessing 20 tasks and 20 trajectories, randomly selected and mixed from both the MM-Traj dataset and the filtered-out data. There are 600 tasks and 600 trajectories in total. Notably, the participants were blinded to the source of the data to ensure unbiased evaluations. Participants rated the quality of tasks and trajectories on a range from 1 to 10, where a higher score indicates better quality. The average scores, as shown in Table A, show that the MM-Traj dataset significantly outperforms the filtered-out data in both task and trajectory quality. These results confirm the effectiveness of our verification process.

Table A: User study for the verification process, where 30 persons were recruited and each one was assigned with 20 tasks and 20 trajectories.

W3. Experimental results on the GTA benchmark are more promising than those on the GAIA benchmark. However, the overall performance of T3-agent is not superior to that of the other agents in comparison. Specifically, on the GAIA benchmark, the HF agent with GPT-4o performs twice as well as the T3-Agent.

R: Compared to the GTA benchmark, the GAIA benchmark is more challenging due to its more complex tasks and longer trajectories, which demand stronger context understanding and reasoning capabilities. Although the HF agent using GPT-4o achieves higher performance than ours, it is important to note that GPT-4o benefits from significantly larger-scale training data and larger model size. In contrast, our agent is built on the MiniCPM-V-8.5B model, an open-source model with 8.5B parameters, fine-tuned on the MM-Traj dataset of 20K samples. The scales are considerably smaller than those of GPT-4o in both model size and data volume, which understandably lead to performance differences. Therefore, directly comparing our agent with agents using GPT-4o is not entirely fair.

To provide more comprehensive and fair comparisons on the GAIA benchmark, we include results from agents built with open-source models of similar sizes, such as LLaVA-NeXT-8B, InterVL2-8B, and Qwen2-VL-7B, as shown in Table B. These comparisons demonstrate the competitive performance of T3-Agent within the open-source ecosystem. We will add the comparisons in a revised version.

Table B: Comparisons on the GAIA benchmarks with open-source models