GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation

We appreciate the reviewer's feedback and address the main concerns below.

W1:

Our current training approach, which leverages ground-truth trajectories, can be viewed as a form of imitation learning. We frame our current work as a first stage toward addressing complex GUI tasks. The primary goal of this stage is to establish a strong foundation by enabling the agent to acquire basic operational skills from a high-quality dataset. Without the guidance of ground-truth trajectories, learning from sparse rewards alone (especially in the early stages) would lead to inefficient reinforcement learning and significantly slower convergence.

We argue that exploring this foundational stage is itself an important and standalone research contribution. It provides a reliable starting point for building agents capable of tackling more complex and realistic GUI tasks.

To address scalability in the later stages, once the model has acquired essential skills through imitation learning, we further refine it using reinforcement learning within a dedicated simulator. In this environment, the agent no longer relies on step-by-step demonstrations; instead, it receives reward signals based solely on the final task outcome.

W2:

We respectfully disagree with the assertion that "the RL stage may not add much value." As shown in Table 1 on the Mind2Web benchmark and Table 2 on the AITW benchmark in the manuscript, our method achieves significant improvements in zero-shot task success rate by 31.5% and 50.7%, respectively. These substantial gains clearly demonstrate the added value of our RL stage beyond supervised fine-tuning.

That said, we agree that performing RL in an online interactive environment is a promising direction for future work, and we consider it complementary to our current setting.

W3:

Thanks for your suggestions, we have added more realistic environmental results in the table below. For WebArena, it utilizes Axtree and SoM annotations to guide action execution, requiring agents to provide element IDs rather than click coordinates. As our vision-based agent produces click coordinates, it is fundamentally incompatible with WebArena’s evaluation framework. Instead, we evaluate our agent on the Android World online benchmark, which comprises 116 tasks across 20 real-world mobile applications. Our agent demonstrates a 16% performance improvement over the prior 2B-parameter state-of-the-art model, InfiGUIAgent. Detailed results are presented in the table below, further confirming its efficacy in complex GUI navigation scenarios.

Additionally, we evaluate our agent on the OS-World benchmark (chrome-split), with findings reported in the supplementary material Table 3. Besides, to align with prior works such as ShowUI, we supplement our MiniWoB experiments under the supervised fine-tuning setting.

Collectively, these outcomes highlight that our approach sustains top-tier performance and robust generalization across diverse, challenging online interactive environments.

Table 4-1. Step success rate (%) of 2B-parameter models on online benchmarks.

W4:

We clarify that in the limitations section of the paper, we specifically referred to online training as a direction for future exploration, not online evaluation.

That said, we fully agree with the reviewer on the importance of evaluating GUI agents in dynamic and realistic environments. To this end, we have already conducted extensive online evaluations across diverse environments, including MiniWoB (Table 4 in the manuscript), OS-World (Table 4 in the supplementary), and Android World (Table 4-1), demonstrating the model’s effectiveness in dynamic and realistic settings.

To avoid such misunderstandings, we will revise the limitations section in the final version to more clearly distinguish between online evaluation and online training.

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1:

The reason we adopt GRPO lies in its training efficiency, as it avoids the computational overhead associated with critic models. In contrast, actor-critic methods like PPO require optimization of both actor and critic models, leading to significantly higher resource consumption.

Regarding DPO, our preliminary experiments revealed its sensitivity to preference pair quality. For instance, using Qwen2-VL-2B, we constructed around 2,000 preference pairs on the Mind2Web dataset. The model trained with these preferences achieved step SR of only 28.5%, 25.1%, and 27.6% on the cross-task, cross-website, and cross-domain splits, respectively, while achieving 68.4% on the training preference pairs itself. This indicates a tendency for DPO to overfit in GUI-based tasks, thus compromising generalization performance.

We appreciate the reviewer’s suggestion to consider alternatives such as DAPO or other methods (as RLOO and REMAX). These methods share a similar principle with GRPO: they aim to obtain reliable advantage estimates without relying on a critic model. In particular, DAPO can be seen as an improvement over GRPO. Adopting these methods is indeed feasible. However, we would like to emphasize that the choice of training algorithm is orthogonal to our main contribution. Our work primarily focuses on proposing a framework that integrates structured reasoning, action prediction, and historical summarization to enhance GUI agent performances. An analysis of such optimization strategies would be a valuable direction for future work.

W2:

Following the reviewers’ suggestion, we additionally trained the larger Qwen2.5-VL-7B model on the training sets of AITW and Mind2Web, and evaluated it on their respective test sets. We ensured a fair comparison with Qwen2.5-VL-7B baseline by using the same training sets, as shown in Tables 3-1 and 3-2. The step success rate results, demonstrate consistent improvements over the baseline by 3% on AITW and 10% on Mind2Web. It further confirming both the effectiveness and scalability of our method.

Table 3-1. Step success rate (%) of Qwen2.5-VL-7B (SFT) and GUI-Rise-7B on the AITW benchmark.

Table 3-2. Performance (%) of Qwen2.5-VL-7B (SFT) and GUI-Rise-7B on the Mind2Web benchmark.

W3 & W4:

We are grateful to the reviewer for their careful attention. These typos will be fully corrected in the final manuscript.

We appreciate the reviewer's feedback and address the main concerns below.

W1:

We respectfully disagree with the claim that our core contributions mainly reside in prompt engineering. In fact, the effectiveness of prompt engineering depends on the model’s inherent capabilities. However, our experiments on Mind2Web demonstrate that smaller models (e.g., 2B/3B parameters) often fail to produce meaningful history summaries or accurate progress estimations. When directly prompting the model to output such information, we frequently observe vague or noisy history summaries, such as: The user started planning a trip; The starting point was set to Braintree with a typo in the "From" field; Although they interacted with some additional elements, their effect on the progress is not explicitly clear. These noisy outputs negatively impact downstream reasoning, and in our experiments, such direct prompting caused a 20% drop in step success rate on the Mind2Web benchmark.

Our key contribution to our method is not in designing handcrafted prompts, but in developing a structured reasoning framework that enables the model to autonomously learn to perform each subtask during training. We view this dynamic subtask learning framework as a core technical innovation that enhances the model’s reasoning capabilities, distinguishing it from conventional prompt engineering approaches.

This design leads to consistent performance gains across both offline and online benchmarks (as shown in Tables 1, 2, and 4 in the manuscript), achieving new state-of-the-art results. These results validate both the practical effectiveness and the technical significance of our method.

W2 & Q1:

We appreciate the reviewers' comments and suggestions for improving the rigor of our statements and address their main concerns as follows.

UI-Hawk reduces visual history images to one-quarter of their original size and adopts a visual token compression ratio of 16, enhancing computational efficiency and performance. However, low-resolution images may lose detailed information, such as small interactive button states, compromising historical data integrity. Furthermore, a key limitation of UI-Hawk is that it has only been evaluated on GUI-Odyssey and its self-proposed FunUI benchmarks. It has not been tested on mainstream benchmarks widely adopted in the GUI navigation field, such as AITW and Mind2Web. Moreover, a direct and fair comparison with UI-Hawk is not feasible, as its results are reported on a privately re-annotated version of GUI-Odyssey (GUI-Odyssey+), with no public code, checkpoints, or benchmark details available.

Regarding Question 1, our statement was intended to address the broader category of methods that rely on history screenshots. Specifically, approaches like UI-Hawk that process these visual sequences inherently face a trade-off between preserving information integrity and computational efficiency. Our agent, GUI-Rise, leverages the history summary capability acquired through training to self-summarize historical task states and executed actions. This approach ensures the integrity of historical information while also reducing computational overhead.

We will expand the discussion of these trade-offs and related methods in the revised Related Work section.

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1:

We appreciate your suggestions and have now included more realistic environmental results in Table 1-1. For WebArena, it employs Axtree and SoM annotations for action execution, requiring agents to output element IDs instead of click coordinates. As our agent is a vision-based model that generates click coordinates, it is inherently incompatible with the evaluation framework of WebArena. Instead, we conduct an evaluation on the Android World online benchmark, which consists of 116 tasks across 20 real-world mobile apps. Our agent achieves a 16% performance improvement over the previous 2B-parameter SOTA model, InfiGUIAgent, further validating its effectiveness in complex GUI navigation scenarios, with results presented in the table below.

Additionally, we evaluate our agent on the OS-World benchmark (chrome-split), with results reported in the supplementary material Table 3. Moreover, to align with prior works (e.g., ShowUI), we supplement our MiniWoB experiments under the supervised fine-tuning setting. Moreover, to align with prior works such as ShowUI, we supplement our MiniWoB experiments under the supervised fine-tuning setting.

Overall, these results collectively demonstrate that our approach maintains state-of-the-art performance and strong generalization across diverse and challenging online interactive environments.

Table 1-1. Step success rate (%) of 2B-parameter models on online benchmarks.

W2:

We agree with the observation that GUI-Rise yields a minor improvement in performance over a vanilla SFT baseline under the in-domain settings of Mind2Web and AITW. Similar trends [1] have been observed in other vision-language domains, such as Vision-Language Navigation, where the performance difference between RL-based methods and SFT remains small in in-domain scenarios. This is largely attributable to the strong fitting capability of SFT to the training data distribution.

The core of our work lies in improving the model's generalization ability in more challenging out-of-distribution (OOD) scenarios that are closer to real-world applications. When the evaluation scenario shifts from standard settings to OOD settings, the performance of vanilla SFT is substantially degraded. In contrast, the unique design of our framework helps the model better understand and adapt to unknown tasks, resulting in performance improvements of 31.5% (Table 1 in the manuscript) and 50.7% (Table 2 in the manuscript). This demonstrates the critical value of our work for building GUI agents that can operate reliably in diverse and dynamic environments.

[1] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Q1:

Our reward design represents a trade-off: we introduce a dedicated reward specifically for the history summary subtask, while only two types of rewards, format reward and action reward, are designed for the other components. This choice balances model performance and training efficiency. Our framework can be readily extended to incorporate separate rewards for progress estimation and decision reasoning. However, we found that doing so significantly increases the computational complexity during training, while yielding only marginal performance gains.

To verify its specific impact in practice, we conducted a small-scale comparative experiment on the AITW benchmark. The experimental results show that if independent rewards are designed for all components, the overall training duration of the model will nearly double, while the resulting improvement in final performance is marginal (about 1.7% improvement). As a trade-off, we chose to retain a separate reward only for the history summary component to balance performance gains against training efficiency.

Q2:

We generate the history summary after the action prediction to ensure that it reflects the most recent action, thereby providing a complete and coherent context for the next step action prediction. Placing the history summary before the action prediction as part of the structured reasoning has two main drawbacks:

(1) The summary at the current step would lack information from the latest action prediction, resulting in an incomplete historical context, which could degrade the performance of subsequent reasoning and decision-making.

(2) The history summary from the previous step is already incorporated as input to the current structured reasoning. Including the current step’s history summary before action prediction would introduce redundant information.

We sincerely appreciate your thoughtful feedback and positive assessment of our work. We will integrate coordinate-based actions in the WebArena evaluation, and include the results in the revised version. We will also report the performance of Qwen2-VL-2B on OSWorld for the completeness of the comparison.