UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

Thank you for your comprehensive review and supportive feedback on our paper. We appreciate your recognition of the novelty of our self-improving framework, the specific contribution of UI-Genie-RM as the first specialized reward model for the GUI agent domain, the superior performance across multiple benchmarks, and the value of our open-source commitment. Below we address specific points raised in your review.

W1: The sufficiency of using five most recent screenshots to process historical context.

A1: We conduct an ablation study analyzing our model's performance and efficiency across different task complexities.

For simple tasks(<5 steps), the actual input is naturally limited by the number of all available history(e.g., 3 screenshots at step 4). We evaluate preserving at most 1, 3, and 5 screenshots in the following Table.1. While using one screenshot is most efficient, it shows limited performance due to insufficient historical context, particularly in outcome reward evaluation. Using up to five screenshots achieves the best performance without introducing significant inference overhead, since the actual number of processed images remains low.

Table.1 Model performance on simple task with different seen images.

For complex tasks(≥8 steps), we evaluate the screenshot number at 5, 8, and 10. We observe that using 5 screenshots achieve comparable performance to 8 screenshots, while increasing to 10 leads to performance degradation, as presented in the following Table.2. This suggests that older screenshots beyond the 5 most recent ones may distract the model by less relevant information. Moreover, larger window sizes significantly increase the computational cost. These results demonstrate the effectiveness of our choice of using five most recent screenshots to achieve the best accuracy and efficiency trade-off.

Table.2 Model performance on complex task with different seen images.

W2&Q1: Clarifications on Section 3.

A2: We apologize for any lack of clarity in Section 3. We provide explanations as follows and will revise our paper accordingly.

Why rule-based verification is needed for constructing data for the reward model? Will it filter out some data from the base GUI operation trajectory datasets?

When constructing the initial training dataset for the reward model, the rule-based verification method is used for getting negative step-level actions. Existing human-annotated GUI agent datasets (AndroidControl, AMEX, etc.) only provide successful trajectories consisting of action sequences, which are used as the positive samples in reward model training dataset. To get the negative samples, we provide the task instruction of GUI agent dataset to our initial agent model, let it sample the step-level actions, then rule-based methods are used to label these step-level actions as negative or positive (Line153-160, rule-based verfication). For example, given a sample click("empty_space") and the corresponding positive action click("Send"), our rules can label the sample click("empty_space") as negative.

Thus, the rule-based methods will not filter out the original ground-truth trajectories, since we treat them as correct positive examples. The rule-based methods is only used to get the step-level negative actions.

In the iterative model fine-tuning, what is the definition of "rounds", "steps"?

Round: The "round" refers to one full cycle of our iterative self-improvement framework, consisting of three phases: 1) Trajectory exploration: The agent generates new trajectories using reward-guided search. 2) Data expansion: Verifying the outcomes of these new trajectories, adding the successful ones to the agent training datasets, and expanding the reward model's dataset with all labeld steps. 3) Fine-tuning: Training both the agent and reward model on these expanded datasets. Based on the above definition, our fine-tuning process thus has three rounds (Line206).

Step: A "step" refers to a single action taken by the agent within a task trajectory. At each step, the agent generates a candidate action, which is then evaluated by our UI-Genie-RM to produce a step-level reward.

UI-Genie-Agent-16k is not mentioned in Section 3 but is it generated during the iterative improvement? How?

Yes, the UI-Genie-Agent-16k dataset is directly generated during the iterative improvement loops (Line239). As detailed in Section 3.3 under "Training Data Expansion", we use UI-Genie-RM for outcome verification to identify successful task completions. These verified, successful trajectories are added to the agent's training set, finally resulting in 16k such trajectories.

W3: Quantifying performance gains of the self-improvement pipeline.

A3: We have studied the performance evolution of both the reward model and agent model in Section A.3 and Figure 1 of our supplementary. The key findings is summarized in the following Table.3. The results demonstrate that the RM's performance continuously improves with each iteration, which in turn boosts the agent's task success rate. This consistent improvement can be attributed to two main factors:

• High-Quality foundation: The reward model is initially trained on high-quality data derived from human-annotated trajectories. In subsequent rounds, we integrate this base data with newly collected data, improving the model through exploring diverse trajectories while maintaining its core capabilities.
• Iterative enhancement: Through the iterative process, the agent model enhances its ability to tackle more complex tasks, generating high-quality reward training samples with increased difficulty. This iterative learning improves the reward model's discriminative ability in challenging scenarios.

Table.3 Model performances after different rounds.

W4: Out-of-domain performance of UI-Genie-RM.

A4: We evaluate UI-Genie-RM on a out-of-domain test set constructed from the AndroidWorld benchmark, using 11 apps that were unseen during UI-Genie-RM training. As shown in the following Table.4, UI-Genie-RM achieves performance competitive with the powerful proprietary model Gemini2.5-pro and significantly outperforms other open-source MLLMs, demonstrating strong generalization to unseen domain. We will add these results to Section 4.4.2 and Table.5 of our paper.

Table.4 OOD performance of UI-Genie-RM.

Q2: Advantage of iteratively improving the reward model.

A: To demonstrate the effectiveness of iterative fine-tuning, we conduct an ablation study where the reward model is not updated after its initial training. As shown in the following Table.5, fixing the RM results in less performance improvement compared with the iterative fine-tuning baseline.The performance gap is most significant in the final round when tackling more complex tasks. This demonstrates that even a relatively small amount of newly generated, high-quality reward data from challenging, dynamic interactions is critical for guiding the agent toward solving harder tasks. We will include these results into our revised paper.

Table.5 Ablation study on reward model fine-tuning.

Thank you for your insightful review and strong support for our work. We greatly appreciate your acknowledgement of the originality of our framework, the significance of automated data generation and the promising results across various models. Below are our detailed responses to the points your raised.

W1: Analysis on the diversity of the generated dataset.

A1: We provide the detailed statistics of the UI-Genie-Agent-16k dataset in the following three tables. We will include these results in our revised paper.

Action Distribution: Our dataset contains 8 distinct action types with a realistic distribution of GUI interactions. While Click is naturally the most frequent action, the significant presence of Type (9.7%) and Swipe (6.7%) ensures the agent learns to handle complex data entry and navigation tasks.

Table.1 Analysis on action distribution.

Goal Diversity: We categorize task goals into 7 distinct categories, showing significant diversity across mobile GUI scenarios. The task goals span a wide range of realistic user intents, preventing model overfitting to a narrow set of tasks.

Table.2 Analysis on goal category.

Task Complexity: Crucially, our dataset emphasizes challenging tasks over trivial simple tasks. We find that medium (5-10 steps) and hard tasks (>10 steps) comprise over 91% of the dataset, which is essential for developing highly capable agents.

Table.3 Analysis on task difficulty.

W2: Lack of some citations.

A2: We appreciate this feedback and will incorporate these relevant work in our revision.

NNetNav proposes an unsupervised approach for training brower agenets by generating synthetic demonstrations through exploration and retroactive trajectory labeling. InSTA presents an internet-scale dataset construction pipeline that use LLMs to automatically generate agentic tasks, complete these tasks, and filter successful trajectories without human supervision. Both work exploit automatic data generation and trajectory filtering mechinisms, aligning with out UI-Genie's self-improving pipeline conceptually. However, our approach addresses mobile GUI tasks with visual information and introduce a specialized reward model UI-Genie-RM for both action-level and task-level trajectory assessments.

W3: Open-source release of framework, datasets, and trained models.

A3: We fully agree with your suggestion regarding open-source releases. To maximize the impact and significance of our contribution, we commit to releasing:

• Complete Framework: The entire UI-Genie framework including data generation, training, and evaluation pipelines.
• Datasets: UI-Genie-RM-517k (reward model training data) and UI-Genie-Agent-16k (agent model training data).
• Trained Models: UI-Genie-RM (reward model) and UI-Genie-Agent in all tested sizes (3B/7B/72B).

W4: Human efforts needed for finding hard-negatives.

A4: Our hard-negative mining process is fully automated and requires no human effort.

After obtaining the initial agent model (trained on existing trajectory datasets, Line153-154 main paper) and reward model (whose training dataset is constructed with rule-based verfication and controlled trajectory corruption techniques), the hard-negative mining process is adopted to find challenging negative examples in reward dataset.

We start with existing annotations from the AMEX dataset which provides task target and ground-truth actions. It also provides functional descriptions for other target-unrelated UI elements. We then use an open-source LLM (DeepSeek-v3) to convert these descriptions into executable actions. These actions are all regarded as step-level negative actions for the task target. After that, we use the initial reward model to score them. If the reward model mistakenly assign a positive score to one action, this action is identified as hard negative. This process is fully automatic without human annotations. We will revise our description in the paper to make this automated pipeline more clear.

Response to "Limitations"

Potential negative impact of fully automated bots for devices

A: Thank you for highlighting this critical consideration. While the UI-Genie framework aims to advance automation in GUI interactions, we acknowledge the importance of recognizing potential risks associated with fully automated bots. If not properly monitored, these bots could lead to misuse, such as conducting financial fraud or stealing private data without authorization. We will expand the Broader Impact section of our paper to include a direct discussion of these risks.

Thank you for your thorough review and constructive feedback provided on our work. We appreciate your recognition of our method's impressive performance, clear writing, and compelling motivation for self-improvement in the costly domain of GUI annotation. Below are our clarifications and responses to the points raised:

W1: Novelty of our paper and the relation to other work like WebRL.

A1: We focus specifically on developing the first reward model tailored for GUI agents, addressing unique challenges in this domain that cannot be directly adapted from other areas. We will add a discussion distinguishing our work from WebRL in the revised paper.

Similarities with WebRL: Both methods can provide outcome rewards for trajectories, enabling the model to learn from both successful and failed attempts.

Key distinctions from WebRL:

• Scalable reward data generation: The ORM training data of WebRL relies on existing WebArena-Lite training datasets with task-specific evaluation functions. In constrast, GUI agents lack such data, and justifying GUI task completion requires combining screenshot information, rather than easily applying task-specific evaluation functions. We propose a suite of deliberately-designed techniques (rule-based verification, controlled trajectory corruption, hard negative mining) to create a large-scale reward dataset from scratch, addressing a fundamental challenge for training GUI RMs.
• Fine-grained process rewards: Unlike WebRL's outcome-only reward, our model provides both task-level and action-level rewards. This fine-grained feedback enables step-wise correctness evaluation and our novel reward-guided beam search, an efficient exploration strategy not present in other work.
• Reward model-guided self-improvement: Different from WebRL, we utilize our reward model not just for trajectory validation, but for actively generating high-quality trajectories without human annotation. Then, our framework refines both reward and agent models iteratively with the increasingly complex training data. In contrast, WebRL's reward model remains fixed after pre-training, limiting its adaptability and preventing continuous generation of new high-quality data to enhance the reward model.

W2: Negative sample construction and human evaluation.

A2: In the negative sample construction process, we have established rigorous rules to evaluate the action alignment effectively, rather than a simple non-GT = negative assumption. Specifically, as discussed in Section B.1 of the supplementary, besides the basic action types, our action space includes a special parameter: action_desc, which describes the target of the action. With the sampled action and action_desc, we write the following code to use some pre-defined rule-based functions and potentially trigger an LLM to judge whether the sampled action is a true negative or equivalent to positive:

def check_action(pred_action, gt_action, task_instruction):
    # return True for positive actions, False for negative actions
    pred_type, pred_param, pred_desc = parse_action(pred_action)
    gt_type, gt_param, gt_desc = parse_action(gt_action)
    
    # Rule-based verification
    if gt_type == 'click':
        # If types and coordinates match, the predicted action is a positive action.
        if pred_type == gt_type and judge_coordinate(pred_param, gt_param): 
            return True           
        # If coordinates differ or type is different,
        # the predicted action might be a positive action (e.g. clicking `+` vs. typing `2`)
        # it can also be negative (e.g. clicking `+` vs. clicking `-`).
        # We thus use LLM to judge action equivalence based on `action_desc`.
        elif pred_type in ['click', 'open', 'type', 'system_button']:       
            return llm_judge_action(task_instruction, gt_desc, pred_desc)        
        else:    
            return False

    # ... other rules for swipe, etc.

We provide the following two examples of sampled actions and corresponding judgement results. We will clarify the details and include the examples in the revised paper.

Table.1 Examples of negative sampling result.

Human evaluation: We manually check 300 random samples each from AndroidControl, AMEX, and AndroidLab. Our above judgement method achieves 92.2% alignment against manual labels, confirming the high quality of our generated reward data.

Table.2 Sample type alignment with human evaluation.

W3: Validation of continuous improvement of the reward model.

A3: We have studied the performance evolution of the reward model in Section A.3 and Figure.1 of our supplementary. The key findings, summarized in the following Table.3, demonstrate that the RM's performance continuously improves with each iteration, which in turn boosts the agent's task success rate. This consistent improvement can be attributed to two main factors:

• High-quality foundation: The reward model is initially trained on high-quality data from human-annotated trajectories, and subsequent rounds integrate base data with newly collected data, maintaining core capabilities while exploring diverse trajectories.
• Iterative enhancement: The iterative process allows the agent model to handle more complex tasks and generate high-quality training samples, thus improving the reward model’s discriminative ability in challenging scenarios.

Table.3 Performances of different round models.

W4: More experimental results and ablation studies.

A4: We provide experimental results of iterative fine-tuning and more ablation studies as follows.

Iterative performance: We provide a detailed iteration-by-iteration analysis in the above Table.3. The results demonstrate substantial and consistent improvement for both models, validating the effectiveness of the self-improvement loop.

More ablation studies: To assess the contributions of each component, we conduct ablations on the following two aspects, as shown in Table. 4:

(1) Removing complex task generation. Without introducing challenging tasks in the final round, the agent’s improvement is limited after Round 2 (31.2→32.6), highlighting the importance of a curriculum of increasing difficulty. (2) Freezing reward model updates. Updating the reward model during the self-improvement phase provides essential guidance for trajectory exploration, resulting in higher-quality synthetic data generation and enhanced performance of both models.

Table.4 More ablation studies.

Discussion with tree search: Our reward-guided beam search provides a diverse set of both successful and informative failed trajectories. This diversity is crucial for iteratively refining our reward model with various positive and negative examples. By contrast, the tree search method is designed to optimize toward a single best solution, pruning other low-reward trajectories. This would prevent us from collecting reward signals from failed trajectories. We will explore adapting tree search-based techniques in future work.

Q1: Equation 1 Typo

A: Equation 1 is not a typo. The trainining objective in Equation 1 is to maximize the log-likelihood of the correct label. For Positive samples $\mathcal{A}_t^{+}$, we want to maximize $\log P(y^{+}|\cdot)$, equavelant to minizing $-\log P(y^{+}|\cdot)$. Similarly, for negative samples $\mathcal{A}\_t^{-}$, we want to maximize $\log P(y^{-}|\cdot)$, equavelant to minizing $-\log P(y^{-}|\cdot)$.

Q2: Avoiding catastrophic forgetting in iterative fine-tuning.

A: We do not implement specialized continual learning techniques. We employ a straightforward and effective data mixing strategy commonly used in VLM training. Specifically, in each fine-tuning round, we combine newly collected data with the data from the previous rounds to train both the reward model and agent model.

Q3: Extension to other GUI domains.

A: Other GUI domains also need scalable data generation and robust reward models. Our methods can create reward datasets and improve agent performance across web and desktop applications. However, this paper focuses specifically on mobile agents, and extending to other environments would require significant computation resources and time for data generation and model training. We consider this a promising direction for future work.

Thank you for your thorough review and positive feedback on our paper. We appreciate your recognition of our motivation, workload, large-scale generated data and open-source commitment. We have carefully considered your valuable suggestions and will address your points below.

W1: The technical innovation of our paper.

A1: While using reward models is a known paradigm, its application to GUI agents is non-trivial and presents distinct challenges. Unlike tasks with verifiable final answers and well-defined reward datasets (e.g. math problems, code generation), GUI tasks involve dynamic screenshots and historical actions, making it difficult to define standard answers, and create large-scale, accurate training data for specialized GUI reward models. Consequently, there lacks a well-established reward model designed for GUI agents that can effectively assess GUI action quality.

Our core innovations directly address these challenges:

• Specialized reward data generation techniques: We develop automated techniques including rule-based verification, controlled trajectory corruption, and hard negative mining to transform existing GUI agent dataset into high-quality reward training data without manual annotation, enabling us to train an initial GUI reward model.
• Co-evolution self-improvement pipeline: We establish a mutual enhancement loop where the agent and reward model continuously improve through dynamic interaction - the agent generates increasingly complex trajectories, the reward model evaluates them, and both model are continuously refined using the generated feedback data along with the initial data.

These contributions provide an efficient, scalable solution for getting the reward model's training data and training both reward and policy models in GUI environments.

W2: Better definition of the newly proposed GUI trajectory reward task.

A2: Thank you for this suggestion. We will add a new Section 3.1 to formally define the task and revise Figure 2 to align with these definitions. The new section will read:

3.1 Task Formulation for GUI Trajectory Reward

Action-level reward: Given a task goal $\mathcal{G}$, current screenshot $\mathcal{I}\_t$, and historical observations $\mathcal{O}\_t={(\mathcal{I}\_0,\mathcal{A}\_0), (\mathcal{I}\_1,\mathcal{A}\_1), ..., (\mathcal{I}\_{t-1},\mathcal{A}\_{t-1})}$, the objective is to evaluate whether a candidate action $\mathcal{A}\_t$ constitutes a correct step toward achieving $\mathcal{G}$. The reward model outputs a positive reward ($y=1$) for correct actions and negative reward ($y=0$) otherwise.

Task-level reward: This task evaluates whether an entire trajectory successfully completes the goal $\mathcal{G}$. The input consists of all actions and resulting screenshots in the trajectory, and the output is a single reward score indicating overall success or failure.

W3: Out-of-distribution test of UI-Genie-RM.

A3: We evaluate UI-Genie-RM on a out-of-distribution test set constructed from the AndroidWorld benchmark, using 11 apps that were unseen during UI-Genie-RM training. As shown in the following Table.1, UI-Genie-RM achieves performance competitive with the powerful proprietary model Gemini2.5-pro and significantly outperforms other open-source MLLMs, demonstrating strong generalization to unseen domain. We will add these results to Section 4.4.2 and Table.5 of our paper.

Table. 1 OOD test of UI-Genie-RM

W4: Ablation studies on three complementary data generation techniques and reward model updating.

A4: The results in the following Table.2 show the impact of three techniques used to construct reward model training data and whether the reward model should be updated in the self-improvement loops. Results of the first four experiments demonstrate each technique provides a significant boost to the reward model's accuracy. The comparision between the fourth and fifth experiments shows that updating the reward model in the self-improvement loops provide a large performance improvement. These results validate the effectiveness of data construction techniques and the design choice of self-improvement framework. We will add these results into the revised paper.

Table.2 Ablation studies on three complementary data generation techniques and reward model updating.

Q1: Necessity of training UI-Genie-RM.

A: To verify the necessity of training and updating a specialized RM, we adopt two variants: (1) using the base Qwen2.5-VL-7B as a fixed RM, and (2) using our initial UI-Genie-RM without updates during self-improvement. As shown in the following Table.3, a general MLLM (Qwen2.5-VL-7B) provides minimal benefit and fails to guide the agent effectively on complex tasks, leading to performance degradation. A specialized designed and fixed in the self-improvement phase reward model is helpful but its effectiveness is relatively limited on complex tasks. Our full framework, where the RM is continuously improved, achieves consistent gains, demonstrating the advantage of a specialized and continuously improving RM. We will include these analysis in the revised paper.

Table.3 Ablation studies on reward models.

Q2: Performance on AndroidWorld benchmark.

A: We evaluate our final agent models on the AndroidWorld benchmark. As shown in the Table.4 below, UI-Genie-Agent-7B achieves a success rate of 36.2%, surpassing strong proprietary models like GPT-4o. Our 72B model shows significant performance gains compared to its base model Qwen2.5-VL-72B, outperforming all other methods. We will incorprate these new evaluation results into our revised paper.

Table.4 Comparisions on AndroidWorld benchmark