Enhancing GUI Agent with Uncertainty-Aware Self-Trained Evaluator

We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:

---

Q1. The biggest concern of this work is insufficient baseline comparisons.

A1. Thanks for your comments. We would like to clarify this point about insufficient baseline comparisons. The prompting-based methods (AgentTrek, WebVoyager, WebJudge, and AutoEval) are non-finetuned models because they use proprietary MLLMs that cannot be trained (as described in Lines 131-133). These prompting-based methods are based on Qwen-VL-Max, and show inferior performances to our URST, demonstrating the effectiveness of URST.

“SFT” adopted in our work is a typical distillation-based method (as described in Lines 248-249). It uses the predictions (CoTs and judgments) from a larger model, Qwen-VL-Max to train Qwen2.5VL-3B for knowledge distillation. “SFT” also achieves inferior results, highlighting the value of our URST. For clarification, we will revise our manuscript and add more detailed descriptions.

---

Q2&3. It is hard to convince that the trajectory evaluator is a must/effective in enhancing GUI agent performance. For section 4.4, the author should also compare their approach with others. In fact, the comparison with baselines here reflects how different approaches are able to improve GUI agent performance, which is the ultimate goal of building a trajectory evaluator.

A2&3. We thank the reviewer for raising this important point about the goal of building a trajectory evaluator. Drawing on prior studies [1-3] and our practical insights, we identify three key roles of the trajectory evaluator for enhancing GUI agents.

1. An autonomous evaluator for online environments

It eliminates the need for manually written verification scripts and has greater scalability and adaptability. The primary contribution of our work is to build this kind of evaluator for assessing the agent-executed trajectories (such as samples in AITW-OOD-traj and AW-OOD-traj), showing the effectiveness and superiority of our URST.

2. A filter for cleaning the noisy dataset

By filtering the incorrect trajectories, we can improve the quality and reliability of the dataset. The impact of the filter increases as the noise level in the data rises. To prove this point, we conducted some comparison experiments by leveraging two settings, Standard-Noise and High-Noise.

• Standard-Noise: Discarding all incorrect trajectories, and using all steps in each trajectory for training GUI agents.

• High-Noise: Discarding all incorrect trajectories, and using the last 40% of steps in each trajectory for training GUI agents. The underlying rationale is that the majority of incorrect trajectories tend to fail in the last few steps. When tasks are highly challenging or involve unseen environments, the data automatically collected by the agent often contains high levels of noise, meaning that the entire trajectory is wrong from the very beginning.

Based on the following Table, we can conclude that 1) our URST can consistently outperform other baselines (including AutoEval* and SFT) in filtering noisy datasets. 2) When the noisy level rises, all filters (based on AutoEval*, SFT, and URST) can achieve a greater impact on agent performance, with URST consistently achieving the best result by a significant margin. 3) URST demonstrates a clear overall advantage, delivering optimal outcomes in time cost, expenditure, and performance.

3. A reward signal for agent inference enhancement

In an online environment, the trajectory evaluator can serve as a reward signal to enhance an existing GUI agent at inference time, using the Reflexion technique [4]. Specifically, a GUI agent first attempts a task, and an external evaluator is used to judge whether its attempt was successful or not. The agent will be prompted to reflect on the failure and retry unsuccessful tasks. We conduct experiments on the most popular online benchmark, AndroidWorld, with M3A agents. The best prompting-based method, AutoEval*, and our URST are used as a reward signal for eliciting reflection. URST achieves superior performance, with a 1.8% improvement over AutoEval*, underscoring its clear advantage in enhancing GUI agents.

[1] Autonomous Evaluation and Refinement of Digital Agents, COLM 2024.

[2] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials, ICLR 2025.

[3] An Illusion of Progress? Assessing the Current State of Web Agents, COLM 2025.

[4] Reflexion: language agents with verbal reinforcement learning, NeurIPS 2023.

---

Q4. Line 123: Why is this approach/formulation restricted to “offline” dataset online?

A4. For training a trajectory evaluator, we need a collected dataset containing GUI navigation trajectories. This navigation dataset can be collected in an online environment. When using this collected dataset, we do not need online interaction, thus it can be treated as an “offline” dataset.

---

Q5. Equation 1: Why does the evaluator need to take in the chain-of-thought c as input?

A5. Thanks for pointing it out. This is a typo, we will revise equation (1) by removing chain-of-thought c from the inputs.

---

Q6. Can you elaborate on how the “Prompting-based Methods” are implemented?

A6. The implementation details of prompting-based methods are described in Figures 5-9 in the Appendix. We also provide the codes for reproduction. For AgentTrek, we follow the same setting in the original paper for evaluating the GUI navigation trajectory. The GPT-4o-mini and fasttext classifiers adopted in AgentTrek are used to filter textual tutorials (for better data collection), instead of GUI navigation trajectories. Thus, we exclude these steps.

Thank you for the detailed response, which addressed most of my comments and I will raise my score to 4. I have some follow-up questions:

• Q1: since the prompting-based baselines use the same base models, do they differ in the prompts they use? or are there other differences?
• Q2: how did you construct the high-noise instances?

We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:

---

Q1. Entropy is an indicator of confidence of evaluator. Although the entropy decreases (Fig. 4), it remains high.

A1. Thanks for your insightful comments. As shown in Figure 4 of the main paper, the entropy of the training data sampled by URST is relatively high (a mean value of 0.4), which can be attributed to the goal of uncertainty-aware sampling: selecting uncertainty data with high entropy. If we measure the confidence of the evaluator, we should refer to the entropy of randomly sampled training data (a mean value of 0.02), which is very low and shows a strong confidence of the trained evaluator.

---

Q2. The performance on standard benchmarks (AITW-ID, AITW-OOD) is 2-3%, which is marginal.

A2. Thanks for your insightful comments. In the following table, we present the results and inference time of prompting-based and fine-tuning-based methods. Compared to SFT, URST achieves significant performance improvements, especially on F1 scores (increases 3.49% - 4.82%). Compared to AutoEval*, URST obtains a slightly smaller performance improvement ( increases 2.64% - 3.53% on F1 score), but it delivers a 2.8x inference speedup. Overall, our URST achieves a well-balanced yet remarkable improvement in both performance and efficiency.

---

Q3. It would be great if the authors can evaluate this on LLM agent benchmarks in general e.g. JudgeBench.

A3. We appreciate the reviewer’s insightful comment regarding the general applicability of our approach beyond GUI agents. While we fully agree that evaluating on broader LLM agent benchmarks such as JudgeBench would further strengthen our claims, conducting such extensive experiments is unfortunately beyond our current time and resource constraints. The evaluation setting and training data collection process of JudgeBench differ from those of our MobileGUIBench.

Nevertheless, to address the reviewer’s concern about generalizability, we have extended URST to a web-based benchmark and conducted additional experiments. Recently proposed AgentRewardBench [1] is the first benchmark to assess the effectiveness of MLLM-based evaluators for evaluating web agents. It contains 1302 web trajectories across 5 benchmarks and 4 agents. Due to the lack of training data, we split AgentRewardBench into two sets, 1047 trajectories for the training set and 255 for the test set, and ensure a similar class distribution.

We evaluate the prompting-based (WebJudge, AutoEval*) and fine-tuning-based (SFT, URST) methods on this web benchmark, and report the results in the following Table. URST is trained under the setting of initial-300 samples, iter. 0-200 samples, iter. 1-200samples, while SFT is trained with the same total number of samples. Our URST achieves the highest performance on AgentRewardBench, reaching 87.06% accuracy and 77.85 F1 score, substantially outperforming the other baselines. The results demonstrate that URST maintains its effectiveness and exhibits strong generalization capabilities in a broader setting.

[1] AGENTREWARDBENCH: Evaluating Automatic Evaluations of Web Agent Trajectories, arxiv 2025.

Dear Reviewer oZh1,

Thank you again for the valuable comments. As the interactive discussion window will close soon, we kindly invite you to read our responses and let us know if you have any further questions. Thank you!

Best regards,

Authors

We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:

---

Q1. Reliance on Evaluator Uncertainty: The method relies heavily on the uncertainty of the evaluator to guide trajectory selection. This casts doubt on whether its uncertainty is a trustworthy signal for trajectory assessment.

A1. Thanks for your insightful comments. We would like to provide a detailed clarification. The uncertainty measurement of the evaluator aims to find the most valuable trajectories for the model, as trajectories with high uncertainty values reflect the limitations or knowledge gaps of the evaluator. The measurement of uncertainty does not involve the trajectory annotations and is irrelevant to the trajectory annotation quality.

Although the evaluator is not trained on any ground-truth data, the initialization phase generates a strong base evaluator by distilling from a proprietary MLLM (generally has around an 80% F1 score). In subsequent iterations, the evaluator can leverage a reward model for judgment assessment, and gradually achieve better performance. As shown in the table of A3, we present a new metric for measuring uncertainty and also achieve a superior performance, demonstrating the effectiveness of our uncertainty-aware sampling.

In summary, the uncertainty is a useful signal for trajectory value assessment, and irrelevant to the trajectory annotation quality. The evaluator can achieve self-improvement with the help of distillation from a proprietary MLLM and guidance from the reward model despite the lack of ground-truth data.

---

Q2. Evaluator Evolution: It would be valuable to track and report the performance trajectory of the evaluator itself over the course of training.

A2. Thanks for your constructive suggestions. As shown in Figure 3 of the main paper, we present the overall performance across multiple iterations. It can be observed that all iterations consistently enhance the evaluator’s performance. However, performing additional iterations yields diminishing gains due to the relatively low quality of the newly sampled training data. In the following table, we also provide the detailed results of multiple iterations. The performance of URST on AITW-ID-traj and AW-OOD-traj gradually increases with iterative training. However, its performance on AITW-OOD-traj peaks at iteration 0 and exhibits fluctuations (primarily in the F1 score) in subsequent iterations. We attribute this behavior to the severe class imbalance in AITW-OOD-traj (success: 14%, fail: 86%), as shown in Table 1 of the Appendix. The F1 score is highly sensitive to even small performance variations in the minority class. We will include these detailed analyses in the revised version.

---

Q3. Alternative Sampling Metrics: Comparing the effectiveness of evaluator uncertainty against alternative metrics would offer a more comprehensive justification.

A3. Thanks for your constructive suggestions. In this work, our uncertainty-aware sampling uses sampling-based entropy to measure the uncertainty of the evaluator, which is inspired by self-consistency methods in LLM [1]. Besides the proposed metric, we also devise an alternative metric, prediction-based entropy, which uses greedy decoding to generate a single judgment of each training input, and computes the probability entropy of the judgment prediction. In the following table, we compare these two metrics with our URST. Sampling-based entropy consistently outperforms prediction-based entropy across all test sets.

Our sampling-based entropy uses temperature sampling and entropy of multiple outputs to measure the uncertainty of the evaluator, which could explore various reasoning paths for answer generation and provide a reliable assessment of the model’s confidence. However, prediction-based entropy uses greedy decoding to generate a single output for entropy computation, and is prone to suffer from the over-confidence issue of LLM [2], thus making less reliable measurement.

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[2] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

---

Q4. Baseline RL Methods: There is no ablation or comparison with GRPO, PPO, or other standard reinforcement learning methods.

A4. Thanks for your constructive suggestions. We would like to provide clarification regarding the comparison with GRPO. SGPO in URST is exactly GRPO without normalization in advantage computing. The experimental comparison of SGPO and GRPO is presented in row “URST” and “w/o normalization removing” of Table 2. Compared to GRPO, SGPO can achieve an improvement of 1.44% in the overall F1 score. We also show the detailed comparison in the following table.

PPO needs an additional critic model for value computing, which requires a more complex training setting and a large dataset [1]. Recent studies [1,2,3] show that GRPO with verifiable rewards is computationally efficient and shows advantages in the CoT-based reasoning scenario, compared to PPO. Thus, we refer to GRPO as a strong RL baseline.

[1] What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret

[2] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale

---

Q5. Generalization Beyond Mobile GUI Tasks: It would be helpful to test the pipeline on other GUI domains, such as desktop or web interfaces.

A5. Thanks for your constructive suggestions. Recently proposed AgentRewardBench [1] is the first benchmark to assess the effectiveness of MLLM-based evaluators for evaluating web agents. It contains 1302 web trajectories across 5 benchmarks and 4 agents. Due to the lack of training data, we split AgentRewardBench into two sets, 1047 trajectories for the training set and 255 for the test set, and ensure a similar class distribution.

We evaluate the prompting-based (WebJudge, AutoEval*) and fine-tuning-based (SFT, URST) methods on this web benchmark, and report the results in the following Table. URST is trained under the setting of initial-300 samples, iter. 0-200 samples, iter. 1-200samples, and SFT is trained with the same total number of samples. Our URST achieves the highest performance on AgentRewardBench, reaching 87.06% accuracy and 77.85% F1 score, substantially outperforming the other baselines. These comparisons underscore the effectiveness and generalization of URST.

[1] AGENTREWARDBENCH: Evaluating Automatic Evaluations of Web Agent Trajectories, arxiv 2025.

Dear Reviewer ktZw,

Thank you again for the valuable comments. Here is the brief summary of our rebuttal:

• Concern about reliance on evaluator uncertainty: We provide clarification on this point, and conclude that the uncertainty serves as a useful signal for trajectory value assessment, and is unrelated to trajectory annotation quality.
• Concern about evaluator evolution: We present detailed results across multiple iterations, and reveal a phenomenon of diminishing marginal returns.
• Concern about alternative sampling metrics: we devise an alternative sampling metric, prediction-based entropy, and compare it with our originally proposed sampling-based entropy, providing a comprehensive justification of our original choice.
• Concern about baseline RL methods: We conduct comparative experiments between GRPO and SGPO, demonstrating the superiority of our proposed SGPO.

Other detailed responses can be found in the Rebuttal. As the interactive discussion window will close soon, we kindly invite you to review our responses and let us know if you have any further questions.

Thank you!

Best regards,

Authors

We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:

---

Q1. The paper relies on a proprietary MLLM to generate the initial training data, which could limit true independence in practice.

A1. We thank the reviewer for raising this important point about dependency on proprietary MLLMs. While our method does leverage a proprietary MLLM for both initial data generation and as a reward model during iterative self-training, this design reflects a pragmatic trade-off to address the absence of ground-truth trajectory annotations. Crucially, the role of the proprietary model diminishes over time: it is only used to score the judgment predictions, and the lightweight MLLM’s parameters are progressively refined using its own diverse CoTs. This mimics successful iterative approaches in other domains (e.g., robotics [1] or NLP [2,3]), where advanced models guide early-stage learning but are phased out as the student model converges.

That said, we fully agree that reducing reliance on proprietary systems is desirable. However, our experiments suggest that URST still advances independence compared to end-to-end reliance on proprietary MLLMs (like prompting-based methods, Webjudgment, AutoEval*, and etc.), as the final evaluator operates without external API calls during deployment. We will add this limitation and direction to the revised version.

[1] Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning, CoRL 2023.

[2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models, Google DeepMind, TMLR 2024.

[3] CREAM: Consistency Regularized Self-Rewarding Language Models, ICLR 2025

---

Q2. URST’s scalability or real-time deployment. More details on training cost, iteration limits.

A2. Thanks for your constructive suggestions. In Table 1 of the main paper, we have presented URST with two different MLLMs, Qwen2VL-2B and Qwen2.5VL-3B. Compared to Qwen2Vl-2B, Qwen2.5VL-3B achieves a superior performance across all three test sets, indicating the scalability of URST.

For real-time deployment, we analyze the inference time of prompting-based methods (WebJudge, AutoEval*) and fine-tuning-based methods (URST) in the following table. In terms of inference efficiency, URST demonstrates a significant advantage over both WebJudge and AutoEval. URST achieves an average inference time of only 2.17 seconds per sample, which is approximately 17.6× faster than WebJudge (38.1 s) and 2.8× faster than AutoEval* (5.99 s). This substantial reduction in computational cost highlights the lightweight design of URST and its suitability for large-scale trajectory evaluation. We will include these results and analyses in the revised version.

We also investigate the training cost of URST and provide detailed results on the training time for all iterations. All experiments were conducted on a server equipped with 4 A100 40GB GPUs. As shown in the table below, the initial phase with SFT requires only a small amount of time, whereas the subsequent iterative training with sampling and SGPO accounts for the majority of the time consumption. We will include these results and analyses in the revised version.

The table above also presents the performance across all iterations. It can be observed that each iteration consistently improves the evaluator’s performance. However, performing additional iterations (> Iter. 1) yields diminishing gains due to the relatively low quality of the newly sampled training data. Considering the computational cost, we limited our experiments to three iterations, which were sufficient to achieve the best performance observed in our study. Therefore, we did not further investigate the upper bound on the number of iterations. We agree that the choice of iteration limits is an important factor, and we believe it can be determined by balancing training cost and performance requirements to reach an optimal trade-off under given computational constraints. We will include these results and analyses in the revised version.

---

Q3. The authors could run multiple trials with different random seeds and report standard deviation or confidence intervals.

A3. Thanks for your constructive suggestions. In the following Table, we present the statistical results of SFT and URST with three different random seeds. URST significantly outperforms SFT across all three test sets. These improvements are statistically meaningful given the non-overlapping confidence intervals, particularly for the F1 score, where URST shows a substantial gain (e.g., +4.82 on AITW-OOD-traj). Moreover, the smaller standard deviations in most metrics indicate that URST produces more stable results across different runs. Overall, these findings demonstrate that URST not only improves performance but also enhances result consistency.

---

Q4. It is suggested to quantify the quality of generated chain-of-thought (CoT). And add visualizations or examples of failure cases.

A4. Thanks for your constructive suggestions. To evaluate the reasoning quality of different models, we conducted pairwise comparisons of their Chain-of-Thought (CoT) outputs by using GPT-4.1. Specifically, URST was compared separately with SFT and AutoEval*, and the winning rates of the latter models were recorded. As shown in the table, SFT outperformed URST in only 32% of the paired cases, while AutoEval* surpassed URST in only 36% of the comparisons. These results indicate that URST consistently maintains a clear advantage over both baselines, as it was judged superior in the majority of cases across both pairwise evaluations. This suggests that URST’s reasoning process is more reliable and better aligned with human preferences than that of the other models.

In section 4.5 and C.2, we have presented the case study of examples where URST succeeds or fails. Here, we would like to restate interesting and insightful findings:

1. When inspecting the success cases of URST, we found that URST has advantages of discerning subtle differences, mitigating hallucinations, and tracking fine-grained transitions (as shown in Figure 5 of the main paper and Figure 2 of the Appendix).
2. When inspecting the failure cases of URST, we found that URST has limitations of insufficient observation, GUI elements misinterpretation, and instruction misunderstanding (as shown in Figure 3 of the Appendix).

Dear Reviewer 3VNW,

Thank you again for the valuable comments. As the interactive discussion window will close soon, we kindly invite you to read our responses and let us know if you have any further questions.

Thank you!

Best regards,

Authors