GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

We thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our work’s importance and are grateful for the detailed suggestions, which we address below point by point.

---

Q1: Many details are missing (e.g., specifics about training loss functions), and no code is provided.

A1:
Thanks for your suggestion. We use the standard cross-entropy loss as the loss function for training the MLLM, and we will add more information about training and model details in our next revision. We make sure the code, model, and data are open-sourced and publicly available.

---

Q2: Experimental baselines are somewhat limited. Suggest evaluating the Qwen series models.

A2:
We agree that broader comparisons help assess generalization across architectures. We additionally conduct the GUI Pre-training with our reflection-related data on Qwen-2.5-VL-7B, and the results are shown below. We can see that, similar to the InternVL-8B baseline, after GUI Pre-training with reflection data, the model's reflection-oriented abilities are effectively improved. We will complete the full training pipeline in our framework (including the offline SFT and online training) with Qwen-2.5-VL-7B and add the corresponding results to our paper.

---

Q3: Why does Action Verification in Figure 3 have two Purposes?

A3:
As described in Section 2.2 and shown in Figure 3, Purpose 1 corresponds to the actual action goal (positive), while Purpose 2 is a plausible but incorrect goal (negative). In this task, the model is only provided with one of these two purposes and needs to determine whether the purpose has been achieved. We will add clarifications about this to avoid misunderstanding.

---

Q4: Why not directly use MLLMs to generate the coordinates?

A4:
We use MLLMs to filter the coordinates generated by the GUI model by visualizing the predicted coordinates with some red dots. However, the MLLMs (e.g., Gemini or GPT-4o) can not directly generate or precisely understand the numerical coordinates, so we use our GUI agent to output the coordinates based on the MLLM-generated action thought as described in Section 2.3.

---

Q5: Could alternative strategies be used to generate more representative failure cases during error data construction?

A5:
Thank you for this thoughtful suggestion. We believe the core concern lies in the representativeness of error cases, particularly regarding (1) real-world failure scenarios and (2) low-probability actions.

First, collecting real-world failure for GUI cases is inherently challenging and costly, as it requires a lot of human supervision to avoid undesirable consequences or safety risks that erroneous actions on real apps could result in. Thus, we adopt a safer and more efficient strategy to utilize existing offline dataset to generate synthetic but plausible errors.
Indeed, one of the main challenges in constructing error cases from existing offline datasets is that they only contain successful trajectories, so we lack the post-state screenshots that would result from actual erroneous actions. This makes realistic error simulation non-trivial. To address this, we prompt the MLLM not only to modify the original goal but also to design the error cases to resemble natural and plausible human mistakes — such as misclicking visually similar elements or misinterpreting UI semantics. While these are not exhaustively representative, they capture common failure patterns and provide useful supervision for learning reflection behaviors.

Second, we clarify that our goal in the Offline SFT stage is not to fix model's common mistakes, but only to proactively activate the model's basic ability to anticipate and detect its own errors. The main goal of this stage is to endow the model with the behaviour to detect and correct errors, and these capabilities are then further refined and expanded in the online reflection tuning stage, where real interaction feedback allows more complex behaviors to emerge.

Looking ahead, we plan to explore the use of generative models or learned world models to simulate a wider range of realistic erroneous actions and resulting states, enabling richer error distributions and more robust error-recovery learning.

---

Q6: Add results for Qwen series models in Table 1b.

A6:
Thanks for your suggestion. We have added the corresponding results. Please see the table in A2.

---

Q7: Table 2 should expand “Filtered BC” as “Filtered Behavior Cloning” at first appearance.

A7:
Thank you for pointing this out — we will revise the text to expand “Filtered BC” as “Filtered Behavior Cloning” upon its first appearance for clarity.

We thank the reviewer for the detailed and thoughtful feedback. We appreciate the recognition of our goal to improve error correction in GUI agents and the suggestions to improve the clarity and focus of the paper. We address each point raised below.

---

Q1: Only one end-to-end benchmark result is shown; lack of ablations makes it hard to assess the impact of individual stages or components.

A1:
We appreciate this insightful suggestion. We agree that providing clear ablations is important to demonstrate the necessity and effectiveness of each proposed training stage. In addition to AndroidWorld, we emphasize that our proposed mobile GUI environment (introduced in Section 2.5.1) also serves as a practical and diverse end-to-end evaluation suite. In Section 3.3 and Table 2, we present an end-to-end evaluation on our environment that verifies the effectiveness of adding reflection-related components at different stages.

To further address the reviewer's suggestion, we have also conducted additional ablations on AndroidWorld, where we systematically compare models with and without reflection-related designs in each stage of our overall pipeline. As shown in the table below, for the three stages of our training pipeline (GUI pre-training, GUI offline SFT, and GUI online training), removing reflection-related design in any stage harms final end-to-end performance, and the reflection-related design in the offline SFT and online stages has larger importance.

These results collectively validate the effectiveness of our GUI-Reflection framework as an integrated whole, and reaffirm our core motivation that reflection and error-correction capabilities should be explicitly injected across all stages of the GUI training pipeline.

---

Q2: Paper may benefit from focusing on fewer methods; some parts could be relegated to the appendix; intermediate results in Tables 1 and 2 make the paper hard to follow.

A2:
Thank you for the suggestion. We will improve the organization and presentation of our paper to enhance clarity. We will move less important content into the appendix and streamline the exposition of our pipeline to highlight the core insights and contributions.
Our framework is a progressive learning process instead of a set of isolated data augmentations. Specifically: 1) In the GUI pre-training stage, the model learns basic reflection-oriented atomic skills. 2) In the GUI offline SFT stage, the model learns to exhibit reflection behaviors. 3) In the online training stage, the model learns to actively reflect and self-correct through real-time interaction, improving generalization and robustness under open-ended execution.

For the experiments on the GUI-Reflection Task Suite in Tables 1 & 2, our goal is to show that regular GUI pre-training significantly weakens reflection-related capabilities. We introduce this task suite to both evaluate and recover such abilities at the pre-training stage. Beyond this, the GUI-Reflection Task Suite serves as a complementary benchmark for general-purpose GUI models. While most existing GUI benchmarks focus on grounding and UI understanding, our tasks emphasize higher-level reasoning skills such as verifying action correctness and reversing errors. These reflection-oriented abilities are crucial for building self-correctable GUI agents and are also useful in broader scenarios—for instance, as a verifier or reward model to supervise intermediate steps.

---

Q3: Effectiveness of the proposed method for "Iterative Online Reflection Tuning"

A3:
Thanks for your constructive suggestions. We additionally include two baselines: 1) standard rejection sampling, and 2) sampling trajectories from a more powerful MLLM (distillation), and compare them with our proposed method in the online stage and evaluate the end-to-end performance on AndroidWorld. We evaluate these alternatives under a controlled setting where the total number of online training samples is kept similar across methods. For the distillation baseline, we use Seed1.5-VL (achieve 62.1 on AndroidWorld) to generate trajectories. The final results are shown in the table below.

We observe that these baselines exhibit the following limitations:

• Standard rejection sampling is inefficient for complex tasks involving one or several key steps in which the model might get stuck. The model may repeatedly fail to discover successful actions on its own. In contrast, our method enables the model to receive direct feedback on critical error steps, including pre-error correction and post-error reflection guidance from an MLLM, resulting in more efficient and focused improvement.
• For distillation, we observe several challenges:
  1. If using prompted agentic methods to produce high-quality trajectories, the cost is prohibitively high, making it infeasible to scale.
  2. Even with strong end-to-end models like Seed1.5-VL, we found that their trajectories lack explicit reflection behavior. Through manual inspection and keyword analysis (e.g., "mistake", "incorrect", "unsuccessful"), we observe almost no signs of self-correction or reflective reasoning, even though Seed1.5-VL is trained on data sources similar to UI-TARS, which include some human-annotated annotations.
  3. The student model design has to be aligned with the teacher model. For example, in our original design, the agent model predicts the thought, action description, and action, but the Seed1.5-VL only predicts the thought and final action. Therefore, we have to redesign the output format and action space of our model to be aligned with the teacher model. However, there still exist some issues where the style of the reasoning thought might be inconsistent with the one in the offline stage.
  4. As a result, the effect of distillation in this case is similar to simply adding more offline SFT data, without the benefits of online reflective feedback. This leads to subpar improvements compared to our iterative reflection tuning.

Overall, these comparisons further support the necessity of our design, where targeted correction and reflection behaviors are injected into the learning loop in a way that neither rejection sampling nor distillation can easily replicate. We will continue to expand experiments in this crucial stage to provide more comprehensive insights in the revised version.

---

Minor Comments

• Q4: Section 2.3 details are sparse
  A4: Thanks for pointing this out. We will improve this section and add a pointer to the appendix where the detailed annotation pipelines are described.

• Q5: "pre-training stage" phrasing is confusing
  A5: Thank you for the feedback. In the GUI agent literature, the term "pre-training" typically refers to the GUI-specific pre-training stage applied on top of a base MLLM, and this usage is common in prior work. To avoid confusion, we will explicitly revise the phrasing to "GUI pre-training" throughout the paper to make this distinction clearer to readers.

• Q6: Terminology "agent-based" vs. "end-to-end" seems unclear
  A6: Thank you for the suggestion — we agree that the terminology can be improved. In the revised version, we will adopt the clearer phrasing of "prompted agents" versus "task-specifically tuned agents", which more accurately reflects the distinction we intend to draw.

• Q7: Line 54 claim about RL not helping reflection lacks citation
  A7: Our statement was specifically intended to describe the behavior we observed within the end-to-end GUI training paradigm. In our experiments, we found that GUI models trained with standard offline SFT on successful trajectories always fail to exhibit reflection or self-correction behaviors (please also see quantitative results in A8); therefore, we state that subsequent RL training alone does not reliably recover these capabilities, as such behaviors can not be obtained by exploration (sampling at a high temperature). This empirical observation was a key motivation for our GUI-Reflection framework.
  We will revise the sentence to make this scope and statement clearer, and we will supplement it with supporting results or qualitative examples to avoid misunderstanding and overgeneralization.

• Q8: Quantify the improvement in error-correction behavior over a relevant baseline on AndroidWorld.
  A8: Thank you for the suggestion. To provide quantification results, we analyze the occurrence of reflection-related behaviors in the model's action thoughts on AndroidWorld. Specifically, we measure how often the model generates thoughts containing keywords indicative of self-reflection or error awareness (e.g., "mistakenly", "unsuccessful", "incorrect", etc.) While this does not directly assess the correctness of reflection, it serves as a proxy signal for the emergence of such behaviors. The ratio of the occurrence of reflection-related behaviors in our GUI-Reflection model is 14.2%, while the baseline without reflection-related content in all stages is 0%.

• Q9: Clarify whether some 8B baselines released recently outperform your model
  A9: Thank you — we will include the discussion of these related concurrent methods in the next version.

We thank the reviewer for the detailed feedback and constructive suggestions. Below, we address each point raised.

---

Q1: Only 8B model is tested; no scaling analysis is provided.

A1:
We have additionally trained InternVL3-78B with our proposed GUI-Reflection framework. The model achieves a success rate of 43.1% on the AndroidWorld benchmark, which is comparable to UI-TARS-72B (46.6%), despite UI-TARS being trained with a significantly larger and more diverse set of trajectories. We also train an InternVL3-78B baseline without the reflection-related designs and it only achieves 31.5%. This result highlights the effectiveness and scalability of our reflection-based training pipeline. We acknowledge that the overall performance of GUI agents is jointly influenced by different factors like capacity of the base model and the scale and diversity of the training data. Investigating these factors in isolation and exploring more systematic scaling trends will be an important direction for future work.

---

Q2: Reflection gains measured on synthetic tasks

A2:
We would like to clarify that the apps used in both our environment and the AndroidWorld benchmark are real, commonly used Android apps, and the tasks are carefully designed to mirror practical, real-life usage scenarios such as calendar scheduling, file management, and document editing. This ensures that our evaluations reflect the model’s effectiveness in realistic contexts, rather than synthetic toy settings. Additionally, for research reproducibility, safety, and better controllability, we choose open-source apps where we have full access to low-level APIs, system states, and internal databases. This enables accurate evaluation and flexible task designs, which are difficult to achieve with closed-source or proprietary applications.
We fully agree that evaluating GUI agents in more diverse and naturalistic settings is important. Extending our environment to include a broader set of real-world apps is an active direction and will be prioritized in future work.

---

Q3: Scope limited to Android mobile; no web/desktop/iOS evidence

A3:
We agree that extending to other platforms is valuable. Our proposed GUI-Reflection framework is platform-agnostic by design, and the core methodology — including the reflection task suite, automatic correction data generation, and iterative online tuning — is applicable to a broad range of GUI environments, including desktop and web interfaces. Due to resource and engineering constraints, our current experiments focus on the Android mobile platform, which offers a practical starting point with a rich set of apps and well-supported emulation tools. Nevertheless, we view this as a first step toward a more general and cross-platform framework, and we plan to extend our method to other platforms such as desktop and web GUI environments in future work.

---

Q4: Eval relies on MLLM judge; fraction of synthetic corrections rejected & observed false-accept rate

A4:
Thank you for your valuable suggestions. In our online reflection tuning process, the MLLM-based judgment and annotation are primarily used in the following parts:
1) An MLLM-based verifier judges whether a task is completed by the agent.
We have 215 task templates in our proposed environment, and most of them (202 out of 215) are evaluated by the programmatic verifier, ensuring correctness in judgment. Only 13 tasks require evaluation by the MLLM-based verifier (Gemini-2.0 Flash) because they involve intermediate results, which are not feasible for a programmatic verifier. For each task template, we provide detailed task guidance (including task descriptions and instructions) to the MLLM verifier to ensure accurate judgment. We randomly sampled 100 MLLM judgment samples and manually checked their correctness, achieving 100% accuracy in this part.
2) An MLLM-based verifier checks the step-wise correctness of the successful trajectories.
For trajectories deemed successful by the programmatic or MLLM-based verifier, we use another MLLM-based verifier (Gemini-2.0 Flash) to assess the correctness of each step. Only steps verified as correct are retained for training. We randomly sampled 200 step-wise samples and manually checked their correctness. The results are summarized below (Note: we denote correct steps as positive here):

The verifier achieves near-perfect Precision (99.5%), ensuring that the quality of the step-wise samples for training is maintained. Note that we can be overly strict here (a lower Recall is acceptable) since the number of discarded samples remains relatively small.

3) An MLLM-based checker identifies the first incorrect step of unsuccessful trajectories
For unsuccessful trajectories, we use an MLLM-based checker (GPT-4o) to identify the first incorrect step. We randomly sampled 200 step-wise samples and manually checked the correctness of the judgment from the MLLM-based checker. The results are summarized below (Note: we denote correct steps as positive here), and we find that the MLLM-based checker ensures high quality in this process:

4) An MLLM annotates the pre- and post-error corrections for incorrect steps.
For incorrect steps in unsuccessful trajectories, we use an MLLM (Gemini-2.5-Pro-Exp) to annotate both pre-error and post-error corrections. We also provide carefully designed task guidance and prompts to help the MLLM generate high-quality annotations. We randomly sampled 200 MLLM-annotated samples and manually checked their correctness. The accuracy of the MLLM annotation is 97.5%. Note that we do not use an additional MLLM-based checker to double-check the annotation, as the MLLM annotator already achieves high accuracy in this task.