Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Question 1: Can Mobile-Agent-v2 be transferred to iOS or other platform?

Response:

• Mobile-Agent adheres to a purely visual solution, making it universally applicable across platforms since Mobile-Agent-v1. Mobile-Agent-v2 continues this approach, allowing it to be transferred to any GUI-based system, such as tablets, PCs, TVs, and automotive systems.

• Specifically, we have implemented it on PCs, facilitating operations such as "downloading papers in the browser" and "modifying font and formatting in Word." The relevant code will also be open-sourced on GitHub recently.

Question 2: How to implement knowledge injection.

Response:

Operation knowledge is often a tutorial for a specific app. It is input into the decision model as part of the prompt:

if add_info != "":
    prompt += f'''
### Hint ###
There are hints to help you complete the user's instructions. The hints are as follows:
{add_info}
'''

where "add_info" is the tutorial.

For example, on TikTok, sharing a video requires clicking the fourth icon on the right. However, the LLM may not have this knowledge. In this case, knowledge can be injected: "The share icon is a white arrow pointing to the right on the right side of the screen." When Mobile-Agent-v2 reaches the step where it needs to click the share icon, it will complete the operation based on the injected knowledge. Since the use of knowledge is determined by the decision agent, other steps are not affected by the presence of the knowledge.

Question 3: Lack of other baselines such as CogAgent and AppAgent.

Response:

CogAgent

CogAgent is a QA model that does not possess the capability to perform concrete operations on real devices through tool invocation. Therefore, our dynamic evaluation framework is not applicable to such QA models.

AppAgent

First, AppAgent requires an additional exploration phase for each app, whereas Mobile-Agent-v2 can perform app operations without the exploration phase. Secondly, AppAgent relies on XML, restricting it to the Android platform. In contrast, Mobile-Agent-v2 uses a purely visual solution, making it platform-independent. Thus, comparing Mobile-Agent-v2 with AppAgent is not fair.

Nevertheless, we manually evaluated the AppAgent and the results are shown in the below table. With the same base model, Mobile-Agent-v2 outperforms AppAgent.

Question 4: Memory unit storage format and retrieval method

Response:

• Memory is stored as natural language descriptions of task-related content from historical screenshots. For example, in the task "Check the weather and write a dressing guide in notes," after opening the weather app, the agent needs to remember the weather information from the screenshot for subsequent use. Before exiting the weather app, the agent will add the weather information to the memory unit in plain text.

• The stored memory is shared among all agents. When the agent needs to input the dressing guide, the decision agent will automatically retrieve the relevant information from the memory during inference.

Question 1: Lack of other baselines such as CogAgent and AppAgent.

Response:

CogAgent

CogAgent is a QA model that does not possess the capability to perform concrete operations on real devices through tool invocation. Therefore, our dynamic evaluation framework is not applicable to such QA models.

AppAgent

First, AppAgent requires an additional exploration phase for each app, whereas Mobile-Agent-v2 can perform app operations without the exploration phase. Secondly, AppAgent relies on XML, restricting it to the Android platform. In contrast, Mobile-Agent-v2 uses a purely visual solution, making it platform-independent. Thus, comparing Mobile-Agent-v2 with AppAgent is not fair.

Nevertheless, we manually evaluated the AppAgent and the results are shown in the below table. With the same base model, Mobile-Agent-v2 outperforms AppAgent.

Question 2: Small and simple evaluation set with unclear evaluation method.

Response:

• Mobile-Agent-v2 uses dynamic evaluation, requiring the agent to directly invoke mobile operation tools and connect to a real device for evaluation. This method is more complex and challenging compared to static app screenshots used in existing work, which only evaluate single-step operations.

• Mobile-Agent-v2's dynamic evaluation involves 20 apps, including both system and third-party apps, across various operating systems and languages. Each app has 4 tasks, with an average of 7 steps per task. Each step undergoes three evaluations: planning, decision, and reflection, totaling 1500+ evaluations. This evaluation scale is the largest among works using dynamic evaluation, covering the most apps and having the most comprehensive task coverage.

• We also received endorsements from Reviewer c2St and Xdcr for our evaluation method: "The paper includes a detailed evaluation across different operating systems, language environments, and applications, providing robust evidence of the system's effectiveness." and "The experimental results on real-world mobile apps look promising." We also recognize the community's lack of a general benchmark based on dynamic evaluation. We are currently working on creating this benchmark and providing a rich set of operation tools to encourage more models to participate in dynamic evaluation.

Question 3: Token cost of usage.

Response:

Assuming an average of 7 steps per task, Mobile-Agent-v2, using a multi-agent architecture, consumes a relatively fixed number of tokens per step. In contrast, a single-agent architecture requires inputting a long sequence of operation history and screenshots with each call, leading to increased token consumption as the number of task steps rises.

Question 4: Efficiency reduction in multi-agent systems.

Response:

• For time overhead, although multiple agents work serialize, there are still phases to parallel. The tables below compare the operation times for the Mobile-Agent single-agent framework and the Mobile-Agent-v2 multi-agent framework. "Screenshot" represents obtaining a screenshot and corresponding image processing from the device, while "Tool Call" represents the use of visual perception tools. It is evident that the increased operation time of the multi-agent framework is acceptable under parallel design.

• For memory overhead, Mobile-Agent-v2 calls the large model via API method, so there is no additional memory overhead.

Mobile-Agent (Single-Agent Framework)

Mobile-Agent-v2 (Multi-Agent Framework)

Question 1: Similar multi-agent architectures have been applied in other scenarios.

Response:

Mobile-Agent-v2 is the first work to use a multi-agent architecture in the mobile domain. The required capabilities of each agent, the tasks they perform, the interaction between agents, and their independent units are significantly different from existing work. Below are the challenges faced by Multi-modal Large Language Models (MLLMs) in the mobile domain and the solutions provided by our framework:

1. Input type and sequence length: In the mobile domain, MLLMs' operation decisions face the problem of multi-image long sequences and interleaved text and images input format, which limits MLLMs' decision accuracy. We propose a planning agent to maintain the stability of input sequence length and format in the single image. This input can improve the context learning of MLLMs.
2. Operation grounding: In the mobile domain, MLLMs need to generate decisions and their locations. Currently, mainstream MLLMs (even GPT-4) lack grounding capabilities. To address this, we included a visual perception tool to assist decision-making. This purely visual solution overcomes the limitations of the operating platform and offers greater versatility.
3. Error gandling in long sequences: In the mobile domain, long sequence operation tasks inevitably lead to errors during intermediate operations. When errors occur, MLLMs often struggle to reflect effectively and correct mistakes. We introduced a reflection agent that uses the MLLMs' multi-image understanding capabilities and contextual learning to judge the accuracy of operations and correct errors.
4. Following historical screen information: In the mobile domain, following and navigating history screen information is difficult due to input length limitations and interleaved text and images. We proposed using an independent memory unit to store this information, significantly enhancing the agent's ability to navigate key information.

Regarding the design of the Mobile-Agent-v2 framework, we note positive feedback from other reviewers, such as c2St and cAKE, who endorsed our approach: "The introduction of a multi-agent system to handle different aspects of mobile device operation is a novel approach"; "Multi-agent framework for mobile device operation is novel in the literature". This confirms the novelty of applying a multi-agent framework in the mobile domain.

Moreover, Mobile-Agent-v2 is a practical framework. Our code, released in the open-source community, has received widespread acclaim, garnered thousands of stars, and has been applied in various real-world scenarios. This practical application stands out among the many works on multi-agent frameworks.

Question 2: The relationship between sequence length and operation type requires further study.

Response:

We have compiled statistics on the relationship between operation sequence length and operation accuracy in the table below.

Here are the conclusions:

1. Minimal impact of sequence length in simple operations: Operations that do not require precise coordinates or parameters, such as "Swipe", "Back", and "Home", are minimally affected by sequence length.
2. Success rate variation in complex operations: For click and stop operations, the success rate is higher at the beginning of the task than in the middle or later stages. This is because early-stage operations usually involve primary pages with better guidance, while advanced pages require the agent to have more robust operational knowledge. However, with the multi-agent architecture, there is no significant decline in the middle and later stages despite the sequence length.

Thank the authors for their rebuttals. I will raise my score to 5. However, the results on the multi-modal scenarios suggest that the performance of the proposed agentic framework can be satisfactory only when it has been applied on GPT-4V. This may limit the generalization of the proposed agent frameworks. In the next version of this paper, I would like to see whether the gap between applying Mobile-agent-V2 on GPT-4V and on other open-source models, especially on small models (less than 7B) can be narrowed down. This is because small models are more practical choice when deploying LLMs on the mobile device.

Question 1: Lack of evaluation in special scenarios, such as unachievable or ambiguous tasks.

Response:

• For tasks that cannot be directly completed through the given instructions, Mobile-Agent-v2 can still attempt to complete them. Due to the high flexibility of Mobile-Agent-v2's operation space, it can simulate almost any operation on a mobile device. Therefore, even if the device does not have the conditions to complete the task, Mobile-Agent-v2 can try to create the conditions through mobile operations. For example, if the task is to open "Facebook" but the app is not installed on the device, Mobile-Agent-v2 will open the app store, search for "Facebook," and click to install it. Then it will return to the home screen and open the app.
• For ambiguous instructions that require explicit user input, Mobile-Agent-v2 can resolve them by expanding the operation space. Mobile-Agent-v2 supports custom operations within the operation space. For example, an operation "seek help" can be added with the description: "Use this operation when you have multiple operation paths to get clearer instructions from the user." When multiple choices can fulfill the instruction's requirement, Mobile-Agent-v2 can determine that the current state is ambiguous and proactively invoke the corresponding operation to obtain user guidance.
• Due to inherent biases in Multi-modal Large Language Models (MLLMs), some operations with strict triggering conditions may be difficult for the MLLMs to use. We are also working on improving the MLLMs‘ contextual learning and operation invocation abilities through alignment training and reinforcement learning.

Question 2: How will the agent operate if the agent lacks operation knowledge?

Response:

• Mobile-Agent-v2 can leverage the operation knowledge within the MLLMs and the multi-agent cooperation to complete complex tasks. Existing mainstream MLLMs, such as GPT-4, Gemini, and Qwen-VL, have operational experience with many apps. Even if the MLLMs lacks knowledge of certain apps, the decision agent can infer possible operations based on page content and the output of the planning agent. Even if errors occur, they can be corrected by the reflection agent. Additionally, operational capabilities can be acquired through training, such as Apple's Ferret-UI.

• We propose knowledge injection to explore whether external operation knowledge can compensate for the agent's operational deficiencies. If a task is too difficult for the MLLMs' internal knowledge, knowledge injection can generate tutorials. Details on knowledge injection will be addressed in the next question.

• If knowledge injection is not used, the agent can acquire operation knowledge through self-exploration. The agent will perform possible operations and use the reflection agent to observe the results, determining whether the operation was correct. The knowledge gained from exploration is then added to an external knowledge base as input for the decision agent. This way, the agent will not need to explore when facing the same task again. We selected samples from the test set with exploration processes and recorded the steps required to complete the task in the table below, where "KI" represents knowledge injection. We can see that the efficiency of the additional exploration process is acceptable. |w/o KI|w/ KI| |-|-| |8.6|6.4|

Question 3: How to implement knowledge injection?

Response:

Operation knowledge is often a tutorial for a specific app. It is input into the decision model as part of the prompt:

if add_info != "":
    prompt += f'''
### Hint ###
There are hints to help you complete the user's instructions. The hints are as follows:
{add_info}
'''

where "add_info" is the tutorial.

For example, on TikTok, sharing a video requires clicking the fourth icon on the right. However, the LLM may not have this knowledge. In this case, knowledge can be injected: "The share icon is a white arrow pointing to the right on the right side of the screen." When Mobile-Agent-v2 reaches the step where it needs to click the share icon, it will complete the operation based on the injected knowledge. Since the use of knowledge is determined by the decision agent, other steps are not affected by the presence of the knowledge.

Question 4: No record of incorrect operations in operation history.

Response:

To simplify the structure of the prompt, we have omitted the situation when an incorrect operation occurs. We save the incorrect operations in another part of the decision agent's inputs:

if error_flag:
    prompt += f'''
### Last operation ###
You previously wanted to perform the operation \"{last_summary}\" on this page and executed the Action \"{last_action}\". But you find that this operation does not meet your expectation. You need to reflect and revise your operation this time.'''

Question 5: Can the framework automatically attribute errors to specific modules or provide error statistics?

Response:

• Thank you for your question. This is an important capability for UI interaction agents. However, since the modules within the framework are interdependent, it is challenging to attribute errors to a specific module solely through global reflection after each operation.

• We manually analyzed the causes of errors in Mobile-Agent-v2 on the evaluation set. The results show that errors can occur at any stage. |Planning Agent|Visual Tool Results|Decision Agent| |-|-|-| |12|10|13|

Therefore, to achieve automated error detection, independent reflections need to be designed for each module. For improvement, we will add reflection on the previous module's results while generating results for each module, preventing errors from propagating between modules. This will not increase latency due to the parallel design. Moreover, errors can be intercepted at the point of occurrence. This will be a key direction for our future work.

I thank authors for their detailed responses to my questions.

While each element of Mobile-Agent-v2 may not be novel in itself, its significance lies in configuring the agentic workflow through multiple agents within the mobile application domain. Therefore, I believe that this paper should be evaluated from an end-user perspective. From this viewpoint, it is essential for the framework to automatically identify and resolve the errors, as highlighted in question 5. Additionally, minimizing the exploration, as discussed in question 2, is crucial for enhancing the user experience. Moreover, regarding question 1, the proposed framework might handle ambiguous tasks, but there is a lack of experimental proof.

As these details crucial to enhancing the user experience are missing in Mobile-Agent-v2, I will maintain my current score. I hope these aspects can be addressed in future version of the paper.

Question 1: Overhead of multi-agent.

Response:

• For time overhead, although multiple agents work serialize, there are still phases to parallel. The tables below compare the operation times for the Mobile-Agent single-agent framework and the Mobile-Agent-v2 multi-agent framework. "Screenshot" represents obtaining a screenshot and corresponding image processing from the device, while "Tool Call" represents the use of visual perception tools. It is evident that the increased operation time of the multi-agent framework is acceptable under parallel design.

• For memory overhead, Mobile-Agent-v2 calls the large model via API method, so there is no additional memory overhead.

Mobile-Agent (Single-Agent Framework)

Mobile-Agent-v2 (Multi-Agent Framework)

Question 2: Can Mobile-Agent-v2 be extended to more complex multi-app scenarios?

Response:

• Yes, Mobile-Agent-v2 can be extended to more complex multi-app scenarios. Mobile-Agent-v2 is a purely visual general framework that is not restricted by app type or operating system, allowing it to be freely extended to any scenario. Mobile-Agent-v2 inherently possesses the capability to plan and make decisions in complex scenarios. Benefiting from the planning agent not being affected by long sequence and image-text interleaved inputs, the decision agent can ensure decision accuracy. Additionally, the memory unit can store key information from previously opened apps' screenshot for use in subsequent operations.

• For extremely complex multi-app scenarios, Mobile-Agent-v2 also supports custom extensions to the operation space. By simply adding the functional description of the operation to the operation space, Mobile-Agent-v2 can perform the operation when needed. For example, if the user needs to use the Mobile-Agent-v2 to crawl the screenshot of the mobile app in batches, the user can add the "screenshot" operation, and request the Mobile-Agent-v2 in the instruction to use the operation to intercept the screen after completing the specified operation task. This extension can effectively improve accuracy without affecting operation efficiency.

Question 3: How does the reflection agent determine if an operation is correct and the success rate of error correction?

Response:

• The Multi-modal Large Language Model (MLLM) itself has multi-image understanding capabilities, which supports multi-image input reflection. While the decision agent outputs an operation, it also outputs the operation intent. The internal contextual reasoning ability of the MLLM can be used to determine whether the operation result meets expectations. Additionally, the MLLM has some operation knowledge, so even if the decision agent's operation intent is inaccurate, it can still make a judgment based on the user's task.

• We have compiled statistics on the success rate and average steps required for the reflection agent's error correction in the table below, where "Reflection SR," "Correction SR," and "Average Steps" respectively represent the success rate of reflection, the success rate of error correction, and the average steps for error correction. The results show that over 60% of operational errors can be successfully corrected, with the cost being less than 2 steps. |Reflection SR|Correction SR|Average Steps| |-|-|-| |94%|62%|1.63|

Thank you very much for your feedback and the time you dedicated to reviewing our paper. We greatly appreciate your insights and are pleased to hear that the clarifications we provided were helpful in addressing your concerns. If you have any further questions or suggestions, please do not hesitate to reach out. We would be happy to discuss them with you.