B-MoCA: Benchmarking Mobile Device Control Agents across Diverse Configurations

Dear reviewer QToU,

Thank you for the time and effort in reviewing our work. We prepared responses to address your concerns one by one. Please feel free to leave further comments to address remaining concerns, if you have any.

---

[W1 - Updates from earlier version]

We highlight several updates compared to the earlier version. Above all, we have extended the number of tasks from 60 to 131 to incorporate a wider range of real-life usages, as the reviewer sGrp praises. Furthermore, we benchmark more diverse baselines, including agents built with open-source LLMs, as the reviewer g7ju remarks. We also provide extensive analyses regarding the additional experimental results, as the reviewer cuoT emphasizes.

---

[W2 - Novelty and impact of our work]

In this work, we have proposed a novel benchmark featuring environmental randomization to reflect diverse device configurations in real usages, which would be a key factor for real-world deployment. We emphasize that our contributions to the community include offering extensive experimental results and detailed analyses. These results include comparisons of the generalization abilities of proprietary LLMs versus state-of-the-art open-source LLMs, performance differences between text-only and multi-modal agents, the impact of using few-shot examples, and the role of environmental diversity in training agents for improved generalization. Furthermore, we emphasize that our findings reinforce strong consensus, aligning closely with what recent benchmarks have remarked. For example, these analyses include a limited multi-modal understanding of state-of-the-art LLM agents [1] and frequent hallucinations regarding task completion [2], offering valuable guidance for future research.

[1] Christopher R., et al. (2024). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents.

[2] Jingxuan C., et al. (2024). SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation.

---

[W3/Q1 - Methods for improving generalization ability]

We clarify that the main focus of our experiments is to provide a benchmark reference. As baselines, we have analyzed several methods for improving generalization abilities, including in-context learning with few-shot examples and fine-tuning parameters using the training dataset collected from diverse environments (please refer to Section 3.2 for a detailed explanation of these baselines). In our analyses, we also examine the effects of few-shot examples for proprietary LLMs and the diversity of environments for training in detail (please see Section 4.3 for more details).

Dear reviewer cuoT,

We appreciate your efforts and time in reviewing our work. Regarding several questions and concerns you have raised, we prepared responses one by one. Please feel free to discuss further to address remaining concerns, if you have any.

---

[W1 - Clarification for baseline with training]

We clarify that the custom agents we explore are trained by using datasets collected in training environments, as you suggested. Our analysis, which examines the effect of the number of environments used for dataset collection on the generalization ability of these agents, is available in Section 4.3.

---

[W2 - Clarification on contributions in our conclusions]

We emphasize that our analyses regard generalization ability over environmental randomization as device configuration changes. These results include comparisons of the generalization abilities across various device settings (e.g., diverse language and device types, on top of unseen apps that AppAgent explored) of proprietary LLMs versus state-of-the-art open-source LLMs, performance differences between text-only and multi-modal agents, the impact of using few-shot examples, and the role of environmental diversity in training agents for improved generalization.

---

[W3 - Rationale behind evaluating open-source models]

The main rationale stems from the several challenges of leveraging closed-source models, such as in fine-tuning, as stated in Section 3.2 (line 245). We find that incorporating training while developing agents can be highly effective in our six selective challenging tasks. As discussed in Section 4.2 (starting from lines 422-423), we observe that these agents show success rates surpassing the agents exploiting proprietary LLMs on several tasks. We believe that this clearly shows that training agents can be a promising way. Furthermore, we analyze the effect of the number of environments used for dataset collection in Section 4.3, providing insights with regard to the generalization ability when developing agents based on open-source models via training.

Dear reviewer g7ju,

We highly value your time and efforts in reviewing our manuscript. We have prepared responses to each concern you have raised. These responses include clarification of the contributions in our findings and the contribution of our work compared to existing benchmarks. Please feel free to leave more comments for further discussion.

---

[W1 - Contribution of B-MoCA]

In this work, we have proposed a novel benchmark featuring environmental randomization to reflect diverse device configurations in real usages, which would be a key factor for real-world deployment. We emphasize that our contributions to the community include offering extensive experimental results and detailed analyses. These results include comparisons of the generalization abilities of proprietary LLMs versus state-of-the-art open-source LLMs, performance differences between text-only and multi-modal agents, the impact of using few-shot examples, and the role of environmental diversity in training agents for improved generalization. Furthermore, we emphasize that our findings reinforce strong consensus, aligning closely with what recent benchmarks have remarked. For example, these analyses include a limited multi-modal understanding of state-of-the-art LLM agents [1] and frequent hallucinations regarding task completion [2], offering valuable guidance for future research.

---

[W2 - Comparison with existing benchmarks]

Thanks for raising an important point. Following your suggestion, we provide a detailed comparison of our benchmark platform with other existing works in the table below. We plan to include this comparison in the revised version of the manuscript.

[1] Christopher R., et al. (2024). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents.

[2] Jingxuan C., et al. (2024). SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation.

[3] Daniel T., et al.,(2021). AndroidEnv: A Reinforcement Learning Platform for Android.

[4] Danyang Z., et al. (2023). Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction.

Dear reviewer sGrp,

We are thankful for your time and efforts in reviewing our manuscript. Regarding several concerns you have raised, we have prepared responses, including additional experiments. We also have clarified the strength of our work compared to existing benchmarks. Please feel free to discuss further to address the remaining concerns, if you have any.

---

[W1 - Comparison with existing benchmarks]

Thanks for raising an important question. While our benchmark is grounded on a commonly used framework defining mobile device control as a sequential decision-making problem, our work serves unique and important points. Because our main focus is on evaluating and developing agents with varying device configurations, our framework features an easily executable platform for configuring diverse environments with differing devices (please refer to supplementary code materials), the training and evaluation pipeline regarding those changes, and detailed analyses of the generalization ability of baseline agents on these varying environments. We provide a detailed comparison of our benchmark platform with other existing works, including Web-Shop [1] and MobileEnv [3].

[1] Shunyu Y, et al. (2022). WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

[2] Daniel T., et al.,(2021). AndroidEnv: A Reinforcement Learning Platform for Android.

[3] Danyang Z., et al. (2023). Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction.

[4] Christopher R., et al. (2024). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents.

[5] Jingxuan C., et al. (2024). SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation.

---

[W2 - More analysis on MLLM agents]

Following your nice suggestion, we additionally conducted an experiment analyzing the limitation of MLLM agents, compared to LLM agents, regarding the time complexity. We measure the average time consumed for the agents to make each decision across all steps in the episodes, solving six challenging representative tasks using one test environment. We provide exact values in the table below. We find that the MLLM agents take longer time compared to LLM agents on average. We note that considering additional methods for addressing the limitation in computational complexity is beyond our main focus, while we believe that our findings provide valuable insights and strong consensus to the community when developing new algorithms.

Dear reviewer zFWV,

We highly appreciate your time and efforts in reviewing our work. We have considered your valuable comments and prepared responses to address your concerns one by one. We have also followed several suggestions and conducted additional experiments.

---

[W1/Q1 - Range of tasks]

Thanks for pointing out an important aspect of our work. While creating tasks, we prioritized building simple but regular tasks, without losing their connection to daily life, where the changes in the UIs associated with the tasks are meaningful to challenge the agents. This is because the main focus of our benchmark is to evaluate and develop agents that can understand and manipulate UI elements in mobile devices on varying device configurations. We believe our experimental results demonstrate that there is considerable room for the state-of-the-art models to handle the existing challenges, indicating the complexity of our current set of tasks.

To reflect real-world scenarios more accurately, we plan to extend the set of tasks to include a broader set of tasks with more applications in the future version. We highlight that we also carefully design our open-source platform so that other users can easily incorporate new tasks.

---

[W2 - Contribution of findings in B-MoCA]

We emphasize that our contributions to the community encompass extensive experimental results and detailed analyses regarding environmental randomization, which is a key factor in real-world deployment. These results include comparisons of the generalization abilities of proprietary LLMs versus state-of-the-art open-source LLMs, performance differences between text-only and multi-modal agents, the impact of using few-shot examples, and the role of environmental diversity in training agents for improved generalization. Furthermore, we emphasize that our findings reinforce strong consensus, aligning closely with what recent benchmarks have remarked. For example, these analyses include a limited multi-modal understanding of state-of-the-art LLM agents [1] and frequent hallucinations regarding task completion [2], offering valuable guidance for future research.

[1] Christopher R., et al. (2024). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents.

[2] Jingxuan C., et al. (2024). SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation.

---

[W3/Q2 - Benchmark reference with state-of-the-art methods]

Thanks for the nice suggestion. To improve our benchmark reference, we have conducted additional experiments with the M3A agent [1]. M3A is currently the state-of-the-art GUI agent, featuring a high understanding of the context while performing tasks by prompting the agent to summarize each step of action and the corresponding effect. We experiment with M3A (a11y tree), following the best-performing version on Android tasks in the original paper. We find that M3A achieves the highest performances among MLLM agents using GPT-4o (either zero-shot or few-shot). The auxiliary history of context helps agents in many tasks even better than the few-shot examples of human expert demonstrations (e.g., Calculator and Wikipedia). However, we observe that the agents using fine-tuned open-source LLMs with the human expert demonstrations (Custom agent using fine-tuned Llama-3-8B) still outperform in some tasks (e.g., Alarm (complex) and Call). Please refer to the table below for the exact success rate values for each task. We plan to include these in our next version of the manuscript.