Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation

Thank you for your valuable feedback and for recognizing the contributions of our work. Your insights are greatly appreciated and will help us further improve the quality of our manuscript.

W1. Weak performance on the GUI-W

1. About Domain Gap.

   • The performance difference between GUI-W (63.08%) and GUI-I (69.20%) benchmarks stems from the domain gap between web and mobile scenarios, as our model was trained on mobile GUI scenarios.

   • We strategically focused on mobile applications because they present many challenges (e.g., frequent interface updates, diverse interaction patterns, and heterogeneous UI layouts). These characteristics make mobile GUI a more challenging and representative test bed for our approach.

2. Cross-domain Generalization and Methodology.

   • Despite this domain-specific training, our model demonstrates strong cross-domain generalization capabilities. As shown in Table 1 in the main text, it achieves superior critic accuracy on the GUI-W benchmark compared to other MLLMs with many more parameters (e.g., Qwen2.5-VL-72B at 60.05%), even though web interfaces differ significantly from mobile UIs in terms of layout structure and action space.

   • Furthermore, our core methodological contributions - the S-GRPO training strategy and reasoning-bootstrapping based data collection pipeline - are domain-agnostic and can be readily adapted to other GUI environments including web and desktop platforms. We'll incorporate this analysis into Section 4.2.1 (Static Evaluation) of the revised manuscript.

3. Additional Validation on a dynamic desktop benchmark OSWorld [1*]. To more thoroughly validate the model's efficiency in desktop scenarios, we have added experiments on the OSWorld benchmark, which is a challenging dynamic benchmark on a real computer environment that contains 369 computer tasks involving real web and desktop apps. We take Qwen 2.5-VL-72B as the action model (baseline), and the results are shown below.

From the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld (50 vs 38) and outperforms the configuration that employs Qwen2.5-VL-7B as the pre-critic (50 vs 43). The results show that our model also exhibits satisfactory generalization ability in desktop and web environments.

[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24.

W2. Real-world GUI Conditions

1. Diversity of training sets. Although dynamic testing was conducted on AndroidWorld, the training data included over 100 real-world apps, covering a wide range of realistic conditions.
2. Real-world static evaluation. In this paper, we introduce a static evaluation benchmark including GUI-I/S/W test datasets (Section 4.1) that covers a broad spectrum of real-world applications and heterogeneous UI layouts. The consistent superiority of our method on this suite of tests provides evidence of its effectiveness.
3. Real-world experiments. Currently, there is no established suitable real-world dynamic benchmark in mobile scenarios. In order to verify the capabilities of our model in real-world scenarios, we added real-device experiments on Mobile-Agent-v2 test set [36]. In this experiment, Qwen2.5-VL-72B serves as the actor agent, while our proposed GUI-Critic-R1 serves as the pre-critic model. We assess the performance on 24 advanced instructions targeting external applications (Table 5/6 in Reference [36]).

The results show that with the assistance of our pre-critic, the agent successfully executed 6 tasks it had previously failed to execute. For instance, while executing “Find a video on Bilibili and give it the triple—like, coin, and favorite,” the agent assumed the task was complete after liking and favoriting, but the critic noticed that the coin had not yet been given and guided the agent model to finish the instruction.

Q1. How is Isimilar(s,s′) computed?

We leverage LLM (Qwen2.5-72B) to compute the semantic similarity between each generated suggestion and its ground-truth annotation (Lines 225–226 and 264). The prompt used for this calculation is given in Appendix Table 4. We will add a detailed explanation of this similarity-scoring procedure, with reference to Eq. 3, in the implementation section of the revised manuscript.

Q2. Critic reduce the need for reasoning within the action model itself?

Thank you for this valuable suggestion. No, introducing a separate critic does not and should not reduce the need for reasoning within the action model itself.

• Clarification: Foremost, we aim to clarify that the introduction of the critic is intended to complement the action model, rather than to supplant its reasoning capabilities.

• Sequential workflow of the system: In our workflow, action model first independently makes decisions, followed by the critic's evaluation and potential feedback (The relevant prompt is provided in Appendix Table 5). If the action model were too simplified and more error-prone, it would lead to frequent critic interventions and multiple revision cycles, impacting the system's efficiency and response time. This is particularly crucial in GUI automation where timely and accurate first-attempt decisions are important for a smooth user experience.

• Advantage: Based on this consideration, both models should maintain their full reasoning capabilities and complement the system at different points. The action model is responsible for generating concrete GUI actions, whereas the critic model focuses on detecting erroneous decisions and providing corrective guidance. This combination of strong initial decision-making and additional verification makes our system more robust and efficient.

Q3. Computational overhead

Following the reviewer's suggestion, we conducted a comprehensive runtime analysis on AndroidWorld benchmark as shown in the table below. The experiments measure task runtime, step counts, and success rates. To ensure fair comparison, all models (except GPT-4o) are deployed with identical hardware resources. The latency for each step includes environment interaction time, model inference time, and action execution time. "Avg Success Steps" denotes the average step number of all successful tasks, and "Avg success latency" represents the average latency of all successful tasks.

The results show that:

1. Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline: We achieve lower average latency (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism.

   • Analysis of Latency Reduction: Although the critic mechanism introduces an additional VLM forward pass at each step, the significant reduction in the total number of steps (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus avoiding time-consuming retry attempts. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.

2. Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models. It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness. In conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.

In conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.

Thank you for your thoughtful review and acknowledging the potential and contribution of our work. We appreciate your insightful comments, which have provided us with an opportunity to refine our manuscript and address critical aspects that will enhance the clarity and impact of our research.

Q1. Experiments on broader scenarios such as desktop application

Our GUI-Critic-R1 is designed to be domain-agnostic and can generalize well across different GUI environments. As suggested by reviewer, to validate the model's effectiveness in broader scenarios such as desktop, we add a new experiment on the OSWorld benchmark, which is a challenging dynamic benchmark on a real computer environment that contains 369 computer tasks involving real web and desktop apps. We take Qwen2.5-VL-72B as the action model (baseline), and the results are shown below.

From the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld (50 vs 38) and outperforms the configuration that employs Qwen 2.5-VL-7B as the pre-critic (50 vs 43). The results show that our model also exhibits satisfactory generalization ability in desktop and web environments.

[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24.

Q2. How efficient the critic is?

1. Comprehensive efficiency analysis. To make the efficiency assessment more comprehensive and credible, we recorded the step number, token number, and the latency required to complete tasks on the AndroidWorld benchmark as follows. "Avg Success Steps" denotes the average step number of all successful tasks, and "Avg success latency" represents the average latency of all successful tasks.

The results clearly show that:

• Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline. We achieve lower average latency (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism. Besides, our GUI-Critic-R1 generates suggestions averaging less than 30 words.
• Analysis of Latency Reduction: Although the critic mechanism introduces an additional VLM forward pass at each step, the significant reduction in the total number of steps (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus avoiding time-consuming retry attempts. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.
• Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models. It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness. In conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.

2. The rationality of EAR (Efficiency Advantage Rate). We believe that EAR effectively reflects whether our model, compared to the baseline, can complete the same tasks with fewer steps. Average execution time and average steps might be significantly skewed by substantial advantages of a small number of tasks. We fully acknowledge the reviewer's perspective that EAR alone is not comprehensive enough. Indeed, EAR, average step length, and average latency each measure model efficiency from different angles, and all these metrics are necessary for a complete evaluation. We will update the revised manuscript with these results.

Q3. Typo

We thank the reviewer for pointing out the issue in our abstract; we will correct this in the manuscript.

We greatly appreciate your recognition of our work and your insightful and constructive feedback. We have carefully addressed each of your concerns and are confident that the resulting revisions will further improve the quality of our study.

W1 & Q1. Latency Overhead

1. Timing budget: Following the reviewer's suggestion, we conducted a comprehensive runtime analysis on AndroidWorld benchmark as shown in the table below. The experiments measure task runtime, step counts, and success rates. To ensure fair comparison, all models (except GPT-4o) are deployed with identical hardware resources. The latency for each step includes environment interaction time, model inference time, and action execution time. "Avg Success Steps" denotes the average step number of all successful tasks, and "Avg success latency" represents the average latency of all successful tasks.

The results show that:

• Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline: We achieve lower average latency (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism.

• Analysis of Latency Reduction: Although the critic mechanism introduces an additional VLM forward pass at each step, the significant reduction in the total number of steps (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus avoiding time-consuming retry attempts. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.

• Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models. It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness.

In conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.

2. End-to-end delay: As shown in the table below, the critic introduces end-to-end latency per GUI action: 6.28s on average, with 7.11s at p95 and 7.26s in the worst case.

3. Screen Resolution Influence: Additionally, we evaluate the inference time cost of our model at different screen resolutions. Specifically, we resize the original screenshot by the corresponding scale factor and evaluate the average inference time per test case on the GUI-I test set. As shown in the following table, we can find higher resolutions lead to increased inference times and the time difference is relatively small. This indicates that the image resolution does not greatly affect the inference delay.

W2 & Q2 Limited Ablation

1. Ablations on S-GRPO Rewards: Due to our oversight, the annotation in the fourth row of Table 4 was incorrect; this row should present the ablation results for $r_s$. We apologize for this error and will correct it in the revised manuscript. As shown in the table, when the $r_s$ term is removed, the model's performance decreases on both the Critic Accuracy and Suggestion Accuracy metrics. Furthermore, we have added ablation experiments for $r_a$ and $r_f$. Removing either of these components leads to a performance drop, as the accuracy reward ($r_a$) and format reward ($r_f$) are both essential reward components for ensuring the reliability of the intra-group advantage estimation for GRPO.

2. Ablations on Group Size: We analyze the impact of group size in the S-GRPO phase in Section 4.3. The group size balances performance with resource utilization, leading to our selection of 6 as optimal (see Figure 3).

W3 & Q3. False-Positive Results

The false-positive (FP) and false-negative (FN) trade-offs are indeed crucial evaluation metrics for assessing a model's ability to accurately identify decision errors without misclassifying correct actions as errors. We evaluated the FP and FN metrics on the GUI-I dataset, and the results are presented in the table below. CA denotes critic accuracy.

As shown, our model achieves a 10.48% reduction in FN and 4.09% reduction in FP relative to GPT-4o, while remaining competitive with other MLLMs.

• Our method significantly reduces the rate - achieving only 8.69% FN (wrongly blocking valid actions or permitting harmful ones), substantially outperforming Qwen-7B (19.44%) and even surpassing GPT-4o (10.24%).

• This proves that our method achieves a strong balance between efficiency and effectiveness in GUI automation, significantly reducing both false rejections of valid operations and false approvals of harmful ones.

Q4. Extend to desktop or web

1. Cross-Domain generalization to web interfaces. While the model was trained on mobile GUI data from GUI-Critic-Train dataset, we extensively evaluated its transferability to web scenarios through the GUI-W task in our GUI-Critic Benchmark. As shown in Table 1 in the main text, our GUI-Critic-R1 achieves the best performance (63.08%) compared to other open source MLLMs, indicating its effective domain adaptation from mobile to web interfaces despite the inherent differences in UI layouts and interaction patterns.

2. Additional validation on a dynamic desktop benchmark OSWorld[1*]. To more thoroughly validate the model's efficiently in desktop scenarios, we add a new experiment on the OSWorld benchmark, which is a challenging dynamic benchmark on a real computer environment that contains 369 computer tasks involving real web and desktop apps. We take Qwen 2.5-VL-72B as the action model (baseline), and the results are shown below.

From the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld (50 vs 38) and outperforms the configuration that employs Qwen 2.5-VL-7B as the pre-critic (50 vs 43). The results show that our model also exhibits satisfactory generalization ability in desktop and web environments.

[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24.

Q5. Release plan

We have submitted the test code in the supplementary materials, and we plan to open-source the model parameters, dataset, and related resources in the future.

Q6. Typos

We thank the reviewer for pointing out the issues in our abstract and references; we will correct these in the manuscript.

We greatly appreciate the insightful and constructive feedback provided. We have carefully addressed each concern below.

W1. Necessity of pre-critic

We agree that improving the policy model can enhance GUI interaction performance. However, pre-operative critic is also necessary in GUI automation.

1. The decoupled relation between policy and critic model and the necessity of our pre-operative critic.

(1). The policy and critic models complement the system in different dimensions.

• Policy generates actions while critic predicts errors and provides suggestions. Both components contribute uniquely to task completion, and the combination of decision-making and safety verification creates our efficient system.
• According to Table 2 of main text and Table 1 of Appendix, upgrading the policy from Qwen2.5-VL-72B to GPT-4o improves success rate from 27.6% to 29.4% on AndroidWorld (using GUI-Critic-R1 as pre-critic). Similarly, enhancing the critic from Qwen2.5-VL-7B to GUI-Critic-R1 improves performance from 20.3% to 27.6% (using Qwen2.5-VL-72B as policy), showing both components' upgrades enhance overall performance.

(2). Pre-operative critic is necessary in GUI automation.

• In dynamic GUI environments, generating accurate decisions is challenging for the GUI agent, leading to high error rates [36 ,27]. While self-reflection strategies fail in complex reasoning tasks [34, 52], an external pre-critic can objectively detect overlooked mistakes and provide corrective suggestions.
• Dynamic evaluation on AndroidWorld (Table 2) and case studies (Figure 4) confirm that the pre-critic can effectively identify errors and provide remedial feedback in GUI automation, demonstrating the necessity of an explicit critic module.

2. Training an effective policy model with GUI-Critic-Train dataset is challenging due to its larger action space versus critic's binary classification. Policies typically need extensive training data (e.g., UI-Tars [26]). Following reviewer's suggestion, we constructed an equally-sized policy training dataset based on GUI-Critic-Train and applied cold-start and GRPO, yielding a policy model Qwen2.5-VL-7B-Policy. We evaluate it on AndroidWorld as follows.

The results clearly show that Qwen2.5-VL-7B-Policy only obtains a small gain after training (14.6% to 15.5%), which indicates that training a good policy model is difficult. Besides, adding our critic to Qwen2.5-VL-7B-Policy improves the performance from 15.5% to 21.5% obviously, demonstrating the complementary roles of policy and critic model and the necessity of critic model.

3. Focused Scope and Proven Effectiveness. Regarding the challenge of predicting outcomes in complex GUI environments, we acknowledge this limitation but emphasize that our critic primarily focuses on identifying obvious errors through static analysis of the GUI state, rather than predicting all possible outcomes. Though not perfect in anticipating consequences, the 7% SR improvement on AndroidWorld (Table 3) validates its value in detecting errors and providing suggestions, enhancing task completion and efficiency. This confirms pre-critic as a promising research direction.

We will incorporate these clarifications in the revised manuscript.

W2. Technical novelty and insights for broader domains

We respectfully disagree with the assessment of limited technical novelty. Instead of simply adapting existing RFT + RL techniques to a specific GUI automation context, we advance and specialize these methods to meet the distinctive demands of the GUI pre-critic task.

1. Introduce technical innovations when compared with the standard GRPO method.

• The GUI pre-critic task presents unique technical challenges that traditional RFT+RL methods cannot directly address, as it requires a more reliable thought process (critique). Specifically, traditional RL rewards only focus on the accuracy of the final answer. In the GUI pre-critic, simply adopting the same technique leads to a critical credit assignment problem: when the model makes a correct judgment, it's difficult to determine whether this resulted from proper reasoning or merely superficial pattern matching. This problem is acute in GUI scenario where reliable reasoning is crucial for both action judgment and suggestion generation.
• To address this challenge, we introduce a suggestion reward as an innovative indirect supervisory signal for CoT reasoning, through measuring the semantic reliability of the generated suggestions. This reward is crucial as producing accurate revision suggestions requires proper analysis of the GUI operation, thereby encouraging the model to develop robust reasoning capabilities. The experiments in Table 4 demonstrate that this simple yet effective innovation of reward raises Suggestion Accuracy (SA) by 4.72 % and 7.02 % on GUI-I and GUI-S, respectively.

2. Introduce two key methodological innovations in reasoning-bootstrapping based data collection pipeline. To address the lack of suitable training data for GUI critics, we propose the following two key modules:

• A progressive Chain-of-Thought paradigm specifically designed for step-by-step GUI operation analysis,
• A Reasoning-Bootstrapping pipeline that enables automatic generation of high-quality critique data

Table 3 in the main text shows that incorporating these two components yields consistent gains, validating their effectiveness.

3. Broader applicability to multimodal interactive domains. Our core methodological contributions are domain-agnostic and can be readily adapted to other online domains.

• The S-GRPO framework naturally extends to scenarios where reliable reasoning and error prevention are crucial, such as robotic manipulation where operational mistakes can be costly.
• Our reasoning-bootstrapping method also provides practical insights for generating high-quality training data in dynamic environments.

W3. Scalability of the data collection process

1. Regarding public datasets concern, our data pipeline supports multiple data sources.

• We support various data. For example, private/custom data from real-world applications, synthetic data generated by automated systems can be used. Public datasets are chosen for their accessibility, quality, representation and diversity.
• We acknowledge that successful trajectories are necessary as ground truth for correct GUI operations. It fundamentally aligns with how pre-critic model should be learned - by referencing successful examples combined with generated negative samples.
• When encountering GUI tasks beyond existing datasets, we can use automated GUI exploration methods [1*][2*] to collect successful trajectories. Negative sampling strategy and rule-based validation criteria can then generate training data, as in Section 3.1. Notably, both modules are domain-agnostic: the former capitalizes on mllm's general understanding of GUI interactions and focuses on universal principles (e.g., visibility and clickability) rather than being tailored to specific GUI scenarios, while the latter applies consistent rules across different GUI contexts.

2. Regarding GPT-4o concern, our GPT-4o filtering is optional, not mandatory.

• As described in Section 3.1, GPT-4o filters erroneous critic annotations for data quality. Without filtering, the ablation study w/o DF in Table 3 reveals only a 2% performance drop and still demonstrates strong performance versus open-source baselines in Table 1. These results indicate that our pipeline's effectiveness isn't entirely GPT-4o dependent.
• GPT-4o's reasoning capability concern is also addressable. The GPT-4o filtering is optional with minimal impact on results. Moreover, GPT-4o is just one possible choice. Our framework can incorporate newer, more capable models as they emerge, and multiple models or verification methods can be combined to enhance reliability.

We will incorporate the aforementioned analysis in the final version of the paper.

[1*] UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

[2*] GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

Q1. Distinct prompts for CoT vs. no-CoT sets

Yes. Different system prompts are used for CoT and no-CoT datasets. CoT dataset uses the prompt in Appendix Table 2, while no-CoT dataset removes thought-related format, requesting only score and suggestion. We'll add this clarification to Section 4.1 and include both prompts in Appendix.

Q2. How do agent use critic output?

The agent treats the critic’s feedback as context and regenerates a new action.

At each step, if critic gives a positive correctness score, the action is executed; otherwise, agent replans with critique and suggestion in context. The feedback integration prompt is in Appendix Table 5. We will add the clarification to Section 4.4.2 in the revised manuscript.

This design leverages complementary roles: policy generates actions while critic detects errors and provides suggestions. Due to GUI environments' dynamic nature, policy alone has high error rates [36]. An external pre-critic offers actionable suggestions, making it essential for reliable GUI automation.

Q3. Typo

We sincerely appreciate your catching this issue. In Table 4, the ✗ in the fourth row should be under $r_s$. We will correct this in the manuscript, with the updated line shown in the W2 response.