GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning

We would like to thank the reviewer‘s constructive feedback. We appreciate the recognition of our simulation engine as a valuable tool for generating and exploring GUI environments. We also acknowledge the reviewer’s concern about the randomized icon placement and its transferability to real-world systems. As discussed in the following section, we attempt to address this issue by considering more structured UI designs in our experiments to maintain a balance between OOD robustness and practical relevance. Additionally, we have provided further clarification on the workings of the engine and its generated environments. The feedback has greatly helped us in refining these aspects.

Below are our detailed responses to Reviewer JzUR.

1. Weakness 1: Shouldn't it be more suited to learn icons with placement that is similar to modern-day UI design to complete its task to be useful? Whether randomized icon placement affects generalization in real-world environments.

• Response:
  • The decision to randomly sample and place icons in the simulated environment is intentional and driven by the goal of promoting generalization. If the icons were not randomized, the agent might memorize which icons to click for specific tasks, making it difficult to evaluate the agent's memorization for the navigation map. Similarly, if the positions were not randomized, the model might overfit to specific data points, memorizing fixed positions and clicking on. These would hamper the evaluation of the agent's grounding ability. By contrast, if the agent is still able to correctly complete tasks in the randomized setting, this would strongly demonstrate that it possesses a solid screen navigation ability.
  • Importantly, we have also validated the agent's performance in real-world GUI environments, where layouts tend to follow more structured, modern UI design patterns. The results showed that the agent can successfully perform screen navigation in well-structured real-world settings. Specifically, we continue training the agents of three methods with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks. As shown in Table 1, MT-RL achieves higher accuracy than ST-RL, which in turn outperforms SFT. This suggests that the agents trained in the simulated environment are able to generalize effectively to real-world benchmarks, encompassing both grounding and interactive tasks.

Table 1: Performance of the agent on real-world grounding benchmark and interactive benchmark.

2. Weakness 2: How the engine works is not clear; for example, what is the generated GUI environment like? how much information and available actions are on the visible page? how many other pages can one page reach on average? Details of the GE-Lab engine and the simulated environment.

• Response:
  • GE-Lab is a novel simulation environment engine. Specifically, GE-Lab first generates a graph of the environment based on user-defined parameters, which determines the navigation relationships between different screens. Then, GE-Lab renders a screen for each node of the graph, with each screen displaying n icons (where n is the degree of the node). Clicking an icon triggers a page transition. The icons used in GE-Lab are randomly sampled from an icon library, enhancing the diversity of the simulation environment.
  • The simulated environment used for our main experiments is Env-Base, as described in Section 4.1. This is a graph with a maximum depth of 7. Excluding system-defined edges representing the home and back actions, the branching structure of the environment follows a node distribution of [5, 3, 2, 2, 1, 1, 0] at each respective level. Figure 6 visualizes this graph structure. The graph is partitioned into five independent subtrees, which are allocated to the SFT training set, RL training set, and test set in a 2:2:1 ratio, respectively.
  • In the visual representation, the number of icons corresponds to the clickable areas that trigger page transitions, with the task completed by selecting the "complete" icon. At the root node, there are 5 icons and clickable areas, allowing access to 5 pages. For first-level child nodes, there are 4 icons (3 regular + 1 "Back") and 4 clickable areas, enabling navigation to 4 pages. For other child nodes, which include "Home" and "Back" icons, the number of icons is NodeNum[i] + 2, resulting in NodeNum[i] + 1 clickable areas, allowing access to NodeNum[i] + other pages. On average, each screen contains more than 5 clickable icons.

3. Question 1: How exactly is the out-of-domain (OOD) test environment constructed? What makes it OOD compared to the in-domain (ID) training set? The construction details of the OOD environment and its differences from the in-domain environment.

• Response:
  • As described in Dataset & Benchmark (A.1, lines 483-487), one of the five subtrees under the root node is excluded from both the training and test sets, representing an OOD environment. The path data from the other two subtrees are used for SFT training, while the remaining two subtrees contribute path data for RL training, which are considered in-domain environments. The OOD samples involve paths that were not encountered during the agent's training process.
  • In addition, we have constructed other more challenging OOD environments. As described in Section 4.1, we generated Env-Image, Env-Name, and Env-Position by augmenting the icon’s image, name, and position, respectively. Furthermore, we introduced noisy icons to create the Env-Noise environment.
  • In terms of the differences between OOD and ID environments, the icons on both types of screens are randomly generated and positioned at different nodes in the graph. As a result, the screen layouts, content, and structure of the two environments are distinct.

4. Question 2: For complex tasks, how long can the history string be? Is there a limit imposed, and if so, how is truncation handled? Context string length constraints and truncation strategies.

• Response:
  • In both multi-turn RL online training and actual inference, we set a maximum number of interaction rounds (e.g., 12). According to our agent protocol (Section 3.2.2), the context token length, composed of the text from a single dialogue round and the current observation image, is less than 1k tokens, which is within the model's context length. Therefore, no additional truncation is required.

5. Question 3: Please provide a formal definition of each component and explain how reward are computed. And what is the weight factor for each component?

• Response:
  • The reward function formula is provided in Appendix A.4. Thank you for your suggestion; we will add a reference to the section containing the details of the reward function in Section 3.3.1.

6. Question 4: RL is known to exhibit high variance. The readers would appreciate it if they can see the stability of the use of RL in this case, which can be demonstrated in Fig. 5 by adding shaded variance regions. The variance in RL training metrics requires visualization to demonstrate training stability.

• Response:
  • Thank you for the reminder. We will include the variance information in Figure 5 in the final version.

7. Nit 2: In Figure 1C, is it true that all pages (the node) have only a degree of 1, i.e., they only have a distinct neighbor? The illustration is not very clear. Illustration of node diagrams in Figure 1(C,D,E).

• Response:
  • To clarify, in Figure 1C, only nodes with degree 1 represent training data that contains knowledge about two connected nodes. This does not imply that every node in the actual state transition graph has a degree of 1.
  • Figure 1C visualizes all training samples as nodes with degree 1, indicating that the SFT training for a single sample only memorizes knowledge of a one-step transition. As shown in Figure 2A, SFT samples follow the data protocol of POMDP, and some of them contain not only one-step transition knowledge (corresponding to degree 1), but also historical trajectories in the history.

8. Nit 3: For Figure 2, I would suggest a top-down, left-to-right ordering of the A, B, and C components. Current ordering makes the visual confusing.

• Response:
  • Thank you for the suggestion. We will adjust the numbering sequence in Figure 2 accordingly.

9. Nit 1 & Nit 4: Typo.

• Response:
  • Thank you for your thorough review. We will make the necessary revisions and perform another round of checks.

10. Paper Formatting Concerns: Reference formatting and authority verification.

• Response:
  • We will thoroughly review all references, confirm their publication status, and ensure the quality of the journals in which they appear. This includes addressing issues related to "Improper citation of foundational RL concepts" and "Formatting errors in in-text citations." Due to space limitations, we will present the final references in the revised version.

Reference

Due to space limitations, please refer to our response to Reviewer DUjd for details regarding the references.

We would like to thank the reviewer for the thoughtful feedback. We appreciate the recognition of GE-Lab as a novel contribution to the field, and we also value the reviewer's acknowledgment of the technical soundness and clarity of our experimental design. Regarding the practical impact, we have now elaborated on how the insights from our work can inform future research and practical applications in the following section. The feedback has been invaluable in helping us refine and strengthen our manuscript. Below are our detailed responses to the Reviewer VsGX.

1. Weakness 1 Quality: This simplification undermines the robustness of the findings for real-world applications.

• Response:
  • To address the concern about whether the simplification of GUI interactions might limit the robustness of our finding, we expanded the action space to assess the agent’s ability to perform a broader range of actions. Specifically, we sampled 1,569 instances from an open-source real-world GUI dataset (including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]) and manually constructed a static benchmark. The test samples in this benchmark cover the following 5 actions: CLICK, COMPLETE, WAIT, SCROLL, and TYPE, the last three of which were not included in the previous action space. We then evaluated the agent without any fine-tuning on this extended benchmark. As shown in Table 1, the agent achieved non-trivial accuracy on WAIT, SCROLL, and TYPE actions, despite not encountering these three new actions during the training process. This indicates that the agent has acquired transferable atomic skills for interacting with GUI elements beyond its original action space, supporting the potential applicability of our approach to more complex real-world scenarios.

Table 1: Performance of the agent on real-world static benchmark with extended action space.

2. Weakness 2 Significance & Question 1: Whether the lack of evidence for transfer learning from GE-Lab to real GUI environments significantly reduces practical impact. & Does training in GE-Lab actually improve agent performance in real GUI environments?

• Response:
  • We further trained the SFT agent, ST-RL agent, and MT-RL agent on real-world datasets (including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]), and the accuracy results are shown in the Table 2. It is evident that the agent's performance on the real-world dataset exceeds the baseline, demonstrating that GE-Lab can improve agent performance in real-world GUI environments. We believe this can address the reviewer’s concern about the practical impact and the major question that the reviewer raised.
  • As shown in Table 2, all agents trained in GE-Lab and then fine-tuned on real-world data outperform both the base model (Qwen2.5-VL-7B-Base) and the model that continued training directly on real-world data without GE-Lab pretraining (Qwen2.5-VL-7B-Continue-Train). Notably, the MT-RL agent achieves the best performance across most benchmarks, with the highest accuracy on ScreenSpot (86.08%), and Refexp (81.77%), as well as strong results on interactive benchmarks like AndroidWorld (15.51%). The average performance of the MT-RL agent is 72.03%, surpassing the other models in overall accuracy, reflecting its robustness. This demonstrates that reinforcement learning in GE-Lab provides generalizable screen navigation capabilities that effectively transfer to complex real-world tasks. The consistent performance gains from SFT to ST-RL to MT-RL, as seen in both the individual benchmarks and the overall average, mirror our simulation findings. These results offer strong empirical support that the skills acquired in GE-Lab are transferable and beneficial when applied to real-world GUI environments, both in grounding and interaction settings.
  • It is worth noting that the results above address both the concern regarding the significance of this work and the major question raised by the reviewer: "Does training in GE-Lab actually improve agent performance in real GUI environments?"

Table 2: Performance of the agent on real-world grounding benchmark and interactive benchmark.

3. Weakness 3 Originality: While GE-Lab is novel, the concept of synthetic navigation environments echoes prior work (e.g., text-based games), and the conclusions (e.g., RL enhances exploration) are not entirely new, reducing the paper’s distinctiveness.

• Response:
  • Although the concept of synthetic navigation environments is not new, we believe that the contribution of this study is distinct in a number of important ways. First, prior synthetic navigation environments have largely been text-based. In contrast, this study is the first to construct a GUI simulation environment that supports interactive training and provides accurate reward signals. This simulated environment enables us to validate the effectiveness of reinforcement learning methods, providing a foundation for future real-world environment development and enhancing agent performance in real-world settings.
  • Second, while it may seem intuitive that reinforcement learning can enhance the exploration capabilities of agents, there are still several practical challenges, such as reward hacking in RL and the lack of progression during MT-RL training. To address these challenges, we mitigated reward hacking by adjusting the training data proportions and improved the efficiency and stability of MT-RL through curriculum learning. The insights gained from addressing these issues in GE-Lab can be effectively transferred to real-world applications.

4. Weakness 4 Clarity: Although generally clear, the paper’s failure to address how its simplified model applies to complex GUI tasks may confuse readers expecting real-world relevance.

• Response:
  • We further investigated the generalizability of our approach by incorporating SFT on a subset of real-world data. Specifically, we augmented our training pipeline (SFT, ST-RL, MT-RL) with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks.
  • The results demonstrate that our training methodology yields agents with strong generalization capabilities across both simulated and real-world GUI tasks. Notably, MT-RL outperforms ST-RL, which in turn outperforms SFT, in terms of accuracy. This observation mirrors the trend observed in our simulated environment as reported in the main text and reinforces our key findings: (1) MT-RL yields better generalization than ST-RL, and (2) ST-RL generalizes better than SFT.
  • We opt to conduct the experiments in simulation primarily due to the significant development cost associated with providing accurate reward signals in real-world environments. Simulation enables us to validate the proposed multi-turn reinforcement learning framework in a controlled manner. This not only demonstrates the feasibility of our approach but also lays a foundation for future development of real-world environments, which would ultimately advance the capabilities of GUI agents.

5. Suggested References:

• We will carefully review the suggested references and incorporate them to enhance our writing and references section.

Reference (Due to space limitations, our reference list includes only the titles of the articles. For reference [9-11], please refer to our response to reviewer DUjd.)

[1] Androidinthewild: A large-scale dataset for android device control[J].

[2] Android in the zoo: Chain-of-action-thought for GUI agents[J].

[3] Amex: Android multi-annotation expo dataset for mobile GUI agents[J].

[4] Mind2web: Towards a generalist agent for the web[J].

[5] Seeclick: Harnessing GUI grounding for advanced visual GUI agents[J].

[6] Os-atlas: A foundation action model for generalist GUI agents[J].

[7] Autogui: Scaling GUI grounding with automatic functionality annotations from llms[J].

[8] A dataset for interactive vision-language navigation with unknown command feasibility[C]

We would like to thank the reviewer for their positive and constructive feedback. We appreciate the recognition of the clarity and significance of our experimental setup, as well as the acknowledgment of the originality of our work in the context of multi-turn RL for GUI agents. We also value the reviewer’s suggestion regarding the analysis of training speed and sample efficiency, and we have now included additional details on this aspect in the following sections of the revised manuscript. The feedback has been invaluable in helping us enhance the comprehensiveness of our work. Below are our detailed responses to the Reviewer htJ2.

1. Question 1: Can you show the difference in training time for the three methods?

• Response: The difference in training time for the three methods.
  • As mentioned in Appendix A.1 (line 478-482), A typical training time for a single SFT agent is approximately 3 to 4 GPU hours. The training of ST-RL agent generally requires about 48 GPU hours, while MT-RL agent typically trains for approximately 36 GPU hours.
  • As stated in Appendix A.1, Dataset & Benchmark, the SFT dataset consists of 60,864 samples, the ST-RL dataset contains 12,439 samples, and the MT-RL dataset is based on online interactive training with 2,162 tasks. The average training time per sample/task is shown in the Table 1 below.

Table 1: Training Time and Sampling Efficiency Comparison for the Three Methods

2. Question 2: Can you show the sampling efficiency difference for the three methods?

• Response: The sampling efficiency difference for the three methods.
  • For both ST-RL and MT-RL, the number of generations per round is 8. Table 1 highlights significant differences in sampling efficiency across the three methods. SFT has the lowest average training time per sample, indicating high efficiency for in-distribution data. In contrast, ST-RL and MT-RL show higher training times per sample or task, reflecting the additional time consumption required by reinforcement learning. However, ST-RL and MT-RL achieve higher interaction counts, suggesting that they are more effective in exploring diverse states and generating robust agent behaviors, particularly in out-of-distribution and interactive settings.

3. Question 3: To improve the clarity, introduce the acronym before using them.

• Response: About acronym.
  • We will carefully review the acronyms in the final version to avoid any potential confusion for the readers.

We would like to thank the reviewer for their thoughtful feedback and positive evaluation of our work. We appreciate the recognition of GE-Lab as a valuable contribution to the field, particularly for its flexibility in modeling complex GUI structures and controllable OOD scenarios. We also thank the reviewer for highlighting the thoroughness of our experimental setup and the clarity of our ablation studies, which helped us assess the performance of different training regimes. The feedback encourages us to further refine our work and explore additional avenues for improving the generalization and interactivity of GUI navigation agents. Below are our detailed responses to the Reviewer DUjd.

1. Weakness 1: Limited environment realism. Whether the simplified simulation environment and action space reduce the generalizability of the findings and constrains the practical implications for deployed agents.

• Response:
  • We acknowledge that the simulated environment used in our study exhibits limited diversity and complexity compared to real-world GUIs. However, we would like to emphasize that evaluating performance on real-world benchmarks is not the primary focus of this work. Instead, our goal is to validate the feasibility of using reinforcement learning for screen navigation tasks. By further adopting real-world icons and layouts inspired by actual apps within GE-Lab, the domain gap can be largely reduced.
  • In fact, the agent trained in the simulated environment demonstrates a notable degree of generalization to real-world GUIs. To assess this, we manually constructed a static benchmark by sampling 1,569 instances from several open-source real-world GUI datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. The test samples cover a broader action space: CLICK, COMPLETE, WAIT, SCROLL, and TYPE. Without any further fine-tuning, we evaluated our agent, which had not encountered WAIT, SCROLL, or TYPE actions during the training process, directly on this benchmark. The results are presented in the Table 1 below. Despite lacking explicit training on WAIT, SCROLL, and TYPE, the agent still achieves non-trivial accuracy on these actions. This suggests that the agent possesses some degree of atomic competence in handling GUI elements beyond its training distribution, highlighting the generalizability of our approach and its practical implications for real-world deployment.

Table 1: Performance of the agent on real-world static benchmark with extended action space.

2. Weakness 2: Missing real-world validation. Although OOD generalization is evaluated within the simulation (Table 3 in paper), the paper does not test the transferability of the trained agents to real-world GUIs or public benchmarks.

• Response:
  • We further investigated the generalizability of our approach by incorporating SFT on a subset of real-world data. Specifically, we continue training our agents (SFT, ST-RL, MT-RL) with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks. The results support the claim that our training framework yields agents with strong generalization capabilities across both simulated and real-world GUI tasks.
  • As shown in Table 2 below, the results demonstrate that our training framework yields agents with strong generalization capabilities across both simulated and real-world GUI tasks. Notably, MT-RL outperforms ST-RL, which in turn outperforms SFT, in terms of accuracy. This observation mirrors the trend observed in our simulated environment as reported in the main text and reinforces our key findings: (1) MT-RL training yields better generalization than ST-RL training, and (2) ST-RL generalizes better than SFT. Additionally，we appreciate review's recommendation to test on the Mind2Web. While Mind2Web is a comprehensive bench for developing and evaluating web agents, its original version is primarily text-based, focusing on HTML and textual descriptions of tasks. Although a multimodal version of Mind2Web, which pairs HTML documents with webpage screenshots, has been released, its core structure and tasks are still fundamentally designed for web agents that operate on textual data. Visual perception of the GUI is a primary input channel in our agent, Therefore, the Mind2Web benchmark does not align with the architectural and data requirements of our multimodal GUI agent.
  • We opt to conduct the experiments in simulation primarily due to the significant development cost associated with providing accurate reward signals in real-world environments. Simulation enables us to validate the proposed MT-RL framework in a controlled manner. This not only demonstrates the feasibility of our approach but also lays a foundation for future development of real-world environments, which would ultimately advance the capabilities of GUI agents.

Table 2: Performance of the agent on real-world grounding benchmark and interactive benchmark.

3. Weakness 3: Incremental novelty of simulation-based training. The paper would benefit from a clearer positioning relative to prior work in this area.

• Response:
  • We have carefully surveyed several existing GUI-based environments—AndroidWorld (116 tasks), OSWORLD (369 tasks), and WebArena (812 tasks)—which target phone-use, computer-use, and browser-use scenarios, respectively. While these platforms serve as valuable benchmarks, the number of tasks in each remains relatively limited. This limitation stems from the significant engineering effort involved in designing tasks that support automated and accurate reward computation — often necessitating task-specific setups such as generating predefined files, which makes large-scale, diverse and scalable training infeasible. To the best of our knowledge, no existing work has demonstrated multi-turn reinforcement learning training in real-world GUI environments, highlighting the practical challenges and the novelty of our simulation-driven approach.
  • In contrast to the above three platforms, GE-Lab, as a simulated environment, is capable of generating an unlimited number of tasks while also automating the provision of accurate reward signals. This is critical for scaling up multi-turn RL training and for exploring learning dynamics that are otherwise infeasible to study in constrained real-world environments.

Reference (Due to space limitations, our reference list includes only the titles of the articles.)

[1] Androidinthewild: A large-scale dataset for android device control[J].

[2] Android in the zoo: Chain-of-action-thought for GUI agents[J].

[3] Amex: Android multi-annotation expo dataset for mobile GUI agents[J].

[4] Mind2web: Towards a generalist agent for the web[J].

[5] Seeclick: Harnessing GUI grounding for advanced visual GUI agents[J].

[6] Os-atlas: A foundation action model for generalist GUI agents[J].

[7] Autogui: Scaling GUI grounding with automatic functionality annotations from llms[J].

[8] A dataset for interactive vision-language navigation with unknown command feasibility[C]

[9] Uibert: Learning generic multimodal representations for ui understanding[J].

[10] Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?[J].

[11] Androidworld: A dynamic benchmarking environment for autonomous agents[J].