DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent

Concern 4. Clarifications over method details of A-RIDE

We have already added more details as suggested to Section 5 and Appendix A.4.

• Do you maintain a separate model for $Q$ and $\rho$? If so, how are those models trained?

  Given your interpretation, since $\rho_t = \dfrac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$ represents the importance sampling ratio between the target policy $\pi$ and the behavior policy $\mu$, we follow the idea that you may be asking about the relationship between the value network (critic) and the policy network (actor). Our framework employs separate models: a ROBERTA-base as the value network for estimating state values and a policy network for generating control actions.

  • The value network estimates the state value function $V(s_t)$, representing the expected return from state $s_t$. Instead of directly regressing on scalar returns, we train the value network to predict the probability that the Monte Carlo return $G_t$ (including both immediate and future rewards) is positive. The critic’s objective is formulated as a binary classification task, with the following loss: $\mathcal{L}(V) = \mathbb{E} \left[ - \mathbb{I}[G_t > 0] \log V(s_t; \theta) - \left(1 - \mathbb{I}[G_t > 0]\right) \log \left(1 - V(s_t; \theta)\right) \right]$, where $V(s_t; \theta)$ is the predicted probability that $G_t > 0$, parameterized by $\theta$, $G_t = \sum_{k=t}^{H} \gamma^{k-t} r_k$ is the Monte Carlo return from timestep $t$, and $\mathbb{I}[G_t > 0]$ is an indicator function, which equals 1 if $G_t > 0$ and 0 otherwise. The value network parameters $\theta$ are optimized by $\theta^* = \arg\min_\theta \mathcal{L}(V; \theta)$

  • The policy network optimizes the following loss: $\mathcal{L} = - \mathbb{E}\_{\mu} \left[ \rho_t A(s_t, a_t) \log \pi(a_t | s_t) \right] - \beta \mathbb{E}\_{\mu} \left[ \mathbb{H}(\pi) \right] + \lambda \mathbb{E}\_{\mu} \left[ \mathcal{P}\_{\text{invalid}} \right]$ Here, $\rho_t$ corrects for off-policy data, $\mathbb{H}(\pi)$ encourages exploration, and $\mathcal{P}_{\text{invalid}}(a_t)$ penalizes invalid actions.

• How is $\mathcal{P}_{\text{invalid}}$ calculated?

  • The invalid action penalty term $\mathcal{P}_{\text{invalid}}$ penalizes actions that the agent cannot execute, addressing the challenge of entropy regularization, which encourages exploration but can lead to invalid actions. In mobile device control tasks, valid actions such as clicking and typing, while invalid commands, such as “rotate screen” in an unsupported context or interacting with non-existent UI elements, waste resources and hinder learning.

    We use Gemini-1.5-Pro to evaluate each action $a_t$. If an action is invalid, $\mathcal{P}\_{\text{invalid}} = 1$; otherwise, it is 0. The penalty term is integrated into the loss as $\mathcal{L}\_{\text{penalty}} = \lambda \cdot \mathbb{E}[\mathcal{P}\_{\text{invalid}}]$, where $\lambda$ controls its impact. This approach ensures exploration remains within valid bounds, enhancing learning efficiency and robustness .

• How is $A(s_t, a_t)$ calculated?

  • The advantage function $A(s_t, a_t)$ is calculated using a one-step advantage estimation approach, as follows $$A(s_t, a_t) = r(s_t, a_t) + \gamma V(s_{t+1}) - V(s_t)$$ Here, the formula comprises two future components: the Monte Carlo return from timestep $t+1$ to the terminal state $s_H$, which is determined by success or failure signals, and hard-coded penalties for repeated actions and certain violations, serving as immediate reward signals. This combination provides a long-term cumulative reward estimate. $V(s_t)$ represents the estimated state value at timestep $t$, $V(s_{t+1})$ is the estimated state value at the next timestep $t+1$, and $\gamma$ is the discount factor.

• Why does DPER improve sample efficiency?

  • DPER improves sample efficiency by prioritizing high-value transitions, such as those with significant TD errors or high importance sampling ratios. This focus enables the agent to learn effectively from previously explored states, reducing unnecessary exploration of less relevant areas. By maintaining policy entropy, DPER balances exploration and exploitation, accelerating convergence with fewer samples.

• Section 5 includes a training loss for the trajectory value function $V_{\text{traj}}$, where is it used?

  • The trajectory value function $V_{\text{traj}}$ estimates the expected return from terminal state $s_H$ in sparse/delayed reward environments. It serves as a baseline by labeling reward signals in collected trajectories (analogous to RLHF reward models) and filters/prioritizes trajectories for training. Through this filtering mechanism, $V_{\text{traj}}$ maintains high-quality training data, leading to more effective policy learning.

We sincerely invite you to consider raising your scores if our responses do address your concerns.

We sincerely appreciate the reviewers' valuable feedback and have addressed each concern in detail below.

Concern1: Paper writing

We acknowledge all feedback regarding our paper's writing style and presentation. In our revised version, we have comprehensively improved the manuscript by:

• Enhancing the clarity of the first two sections
• Refining our use of bold text throughout the paper
• Implementing all suggested writing improvements

We are committed to delivering a more polished and professional manuscript that better communicates our research contributions.

Concern2: Why did we say "DistRL is the first work to scale autonomous, online RL fine-tuning for mobile device control in a distributed environment."

We appreciate the reviewer’s feedback and have worked to clarify and substantiate this claim in our revised version.

Our claim that "DistRL is the first work to scale autonomous, online RL fine-tuning for mobile device control in a distributed environment" is grounded in extensive testing and rigorous reproduction of baseline methods. While DigiRL [3] nominally includes distributed data collection, it has significant limitations in terms of scalability. Our empirical analysis reveals that DigiRL’s implementation suffers from critical issues, including severe inefficiencies in training, incomplete implementation, resource contention when running multiple emulators, and ineffective use of distributed computing resources, all of which hinder its scalability for any Online deployments. In contrast, DistRL is explicitly designed with scalability in mind, incorporating optimizations for distributed architectures such as efficient resource allocation, load balancing, and robust training across multiple machines. This allows DistRL to provide a truly scalable solution for autonomous, online RL fine-tuning on mobile devices. The primary distinction lies not merely in having distributed functionality but in the effective and efficient use of distributed resources, enabling training stability and high performance at scale.

Concern 3. Figure 5(b) needs more explanations; what does "Proportion Above 60%" mean?

• The “Proportion Above 60%” refers to the proportion of recorded time steps within each interval where the agent achieves a success rate over 60%, with “Proportion Above 80%” indicating time steps exceeding 80%. These metrics highlight learning efficiency and stability. As shown in Figure 5(b), DistRL consistently achieves higher proportions than DigiRL[3], demonstrating faster convergence and improved stability due to its asynchronous architecture and optimized data collection and sampling strategies. We have clarified this in the figure caption and main text.

First of all, we really acknowledge the reviewer’s recognition over our works and address each concern in detail below.

1. Concerns regarding the performance discrepancy between DigiRL(single) and DigiRL(Multi):

In the DigiRL (Single) setup, we strictly adhere to the original DigiRL baseline by running up to 32 emulators on a single host machine. However, we observed that implementation limitations prevent all emulators from operating simultaneously at full capacity, even on high-end hardware. This configuration employs synchronous training without optimization, leading to substantial resource contention and performance bottlenecks due to all emulators competing for the same computational resources. We use this setup as a baseline to highlight the limitations inherent in single-machine environments.

In contrast, DigiRL (Multi) distributes the 32 emulators across multiple host machines, with each machine managing a manageable subset of emulators. While maintaining synchronous training, this approach effectively balances the computational load and incorporates minor performance optimizations. By leveraging a distributed architecture, DigiRL (Multi) mitigates the resource contention seen in the single-machine setup, resulting in better resource utilization and improved performance.

2. Validation of the VLM Evaluator (Gemini) Using the Entire Trajectory

We appreciate the reviewer’s concern regarding the effectiveness of using the current observation versus the entire trajectory for success validation. In our experiments, we used the last screenshot along with the last two actions as input to the VLM Evaluator (Gemini) for prompting. This approach provides a concise yet informative context for the evaluator. Please see our supplementary experiments as we replied to Reviewer vpis, we tested the correlation between our autonomous evaluator and human judgments for different agents on the whole General subsets from AiTW:

Our findings indicate that incorporating additional context, such as longer trajectory information, negatively impacts evaluation accuracy. Specifically, the discrepancy between the evaluator's output and human assessments was less than 2% when using only the last image and the last two actions. However, providing more context led to performance drops, significantly higher computational costs, and increased usage of evaluator LLMs, thereby straining our budget.

In fact, empirical tests show that large models like Gemini tend to classify tasks as successful by identifying evidence of success in provided screenshots. The more information Gemini receives, the higher the probability it judges the task as successful. In few-shot scenarios with seven or eight images, adding more tokens causes token explosion, leading to hallucinations. Our algorithm leverages score assignments based on final steps and states, achieving a balance between computational efficiency and evaluation accuracy. We have updated the manuscript with these results and detailed analysis in section 6.3, demonstrating that Gemini provides reliable success validation with minimal input, which is essential for efficient and cost-effective training.

3. Clarification on "On-device End-to-End Agent Performance Evaluation"

Thanks for highlighting the potential confusion in the section title. To avoid misinterpretation, we renamed the section to "Performance Evaluation of Agents Trained with DistRL." This new title more accurately reflects the content, focusing on evaluating the performance of agents trained using our distributed framework across various tasks and environments. We have also updated the text to clarify that while the agents are capable of running on-device due to their compact size, the evaluations were conducted using host machines with emulators for consistency and scalability.

Additional Comments

We are grateful for the reviewer’s suggestions regarding future work. We have dedicatedly revised the paper based on all these valuable comments. Exploring frameworks like SeedRL to further scale our approach is indeed a promising direction. Additionally, we acknowledge the value of evaluating our methods on online benchmarks such as AndroidWorld for standardized comparisons. We plan to incorporate these considerations in our future research to enhance the reproducibility and impact of our work.

We sincerely invite you to consider raising your score given your positive reviews.

We greatly appreciate the reviewers' insightful suggestions and helpful questions, which will enhance our paper's quality. We address each question individually below:

Q1: Clarification on Model Usage and Implementation Details

In practice, we did not encounter any conflicts during the implementation of our system, and there were no intentional changes in model usage.

In our system’s implementation, we maintained a consistent and conflict-free approach by exclusively utilizing the T5-based Multimodal Large Language Model (MLLM) architecture (not vanilla pretrained T5, T5-Base normally has 220M, Our Agent (AutoUI-driven) has 1.3B). This encoder-decoder framework facilitates the seamless integration of various agents, enhancing the model’s versatility and performance, which is aligned with AutoUI [42] design.

Our T5-based MLLM architecture is inherently designed to support diverse decoder initializations, including the integration of pretrained decoder models. While previous works like AutoUI [42] have utilized their own model weights, we could not directly use AutoUI’s model weights to initialize our models because they were trained with AiTW knowledge. Incorporating these weights could introduce unintended biases specific to their tasks as well as prior knowledge to our training and testing data set, which might reduce the credibility and confidence of our system.

Meanwhile, our empirical analysis showed that decoder layers require more diverse initialization patterns than encoder layers to achieve optimal performance in UI-specific tasks, GPT-2 help more than the pre-trained T5 weights. Therefore, the best approach was to manually select a pretrained decoder model to initialize the weights of our T5-based decoder. We chose to initialize the decoder with GPT-2 weights, as this allowed us to leverage GPT-2’s pre-trained language generation capabilities, providing a foundational understanding of language that we could further refine through fine-tuning tailored to our specific tasks.

We did not specify the decoder type/weights in the original explanation because we found that fine-tuning efficiency was already satisfactory with GPT-2. We acknowledge that there might be better options available; however, we adopted this approach simply by drawing inspiration from our baseline DigiRL [3], which also used GPT-2 to initialize their decoders (This is a trick here). By following a structured and methodical approach from existing baselines, we ensured architectural compatibility and effective integration of the GPT-2 weights into our T5-based decoder.

For clarity, refer to our submitted code: https://anonymous.4open.science/r/DistRL-C41F/distrl/models, particularly ./models/model1.py lines 16-30 and ./models/autoui_agent.py lines 20-54. We apologize for any misunderstandings caused by our statements and have already made the statements clearer in the revised version.

Q3: Clarification on Training and Testing Sets, and Generalization Issues

We ensured no overlap between training and testing sets by proper splitting during evaluations. Training sets were selected and formed based on empirical results and iterative trial-and-error, focusing on areas where pure AiTW is limited. For example:

• AiTW tends to result in Chrome-based operations.
• AndroidWorld includes general app controls like "create contact number."
• Expert-added tasks involve more detailed operations, such as checking specific app updates in the app store.

It’s worth mentioning that we did not use any traced data from AiTW, only their task instructions. Training and testing sets examples are provided in Appendix A.5.2, Tables 4 and 5. We will open-source these sets post-review.

Q4: Performance Evaluation During Training and End-to-End Training Considerations

Our primary contribution is improved training performance. We evaluated the trained DistRL agent independently with no overlaps between training and testing sets. The only performance evaluation we conducted was after the training period; we evaluated the trained agent produced by DistRL independently, as reported in Section 6.5. However, we are pleased to provide additional results as requested. During training, we can frequently evaluate checkpoints on the test set. Results averaged in 3 trials are presented here (tested on “General” from AiTW as example):

These additional results with more details were reported in the revised version in Appendix A.6.3 to offer more insights into the agent's performance over time. The reviewer can also check here: https://anonymous.4open.science/r/DistRL-C41F/Rebuttal_extra_exps/TestCurve-Rebuttal.png. We have also included these results in Appendix A.6.3 in the revised paper.

Q5: Suggestion to Plot Performance Curves Against the Number of Environment Interactions Instead of Time

We appreciate the suggestion; however, in distributed RL systems, plotting against time is standard due to asynchronous processes (e.g., IMPALA [6], Figure 6). In fact, comparing on a 'per step' basis may not provide the most balanced perspective in this context because:

• DigiRL-Multi operates in a purely synchronous setting. For each model training step on the learner side, it waits for a fixed number of collected trajectories. This results in a trend that is a straight line with the same scale as the buffer size.
• Our design collects trajectories much faster than the baselines but with definitely fewer trajectories collected "per step" as measured by the learned models.

We also provided experimental results as per the reviewer's request within the same time budgets here:https://anonymous.4open.science/r/DistRL-C41F/Rebuttal_extra_exps/Figure5(c)-Rebuttal.pdf. These show that, given the same time, our system achieves higher collection speeds while maintaining input training information.

We sincerely invite you to consider raising your scores if our responses do address your concerns

Q2: Concerns Regarding Novelty

We have revised Section 2.3 and Section 5 to ensure the organization is excellent and the coverage is thorough.

A. Concerns Regarding IMPALA and IMPACT

As discussed in section 2.3, though old, IMPALA and IMPACT are well-known distributed reinforcement learning algorithms; however, the original work were not directly used as baselines to several key reasons:

1. Inadequate Handling of Fluctuating Online Experiences: Both IMPALA and IMPACT were designed with environments in mind that are relatively stable and do not exhibit the high variability found in mobile device interactions.
2. Lack of efficient buffer management for highly distributed cases.
3. Less supports from off-policy RL Models. It leads to very poor training performance.
4. Scalability and System Design Differences: IMPALA and IMPACT do not offer the same level of scalability and system optimizations required for distributed mobile control.

However, though non-trivial, effective implementation upon IMPALA is doable, here comes our work. Specifically, managing communication, scheduling worker roles, and handling queues and replay buffers demand substantial effort and innovative design solutions, e.g.:

• split FIFO queues into chunks and build circular buffers to support distributed use.
• Resolve thread and process locks and conflicts.
• Optimize memory usage of buffers.
• tailored off-policy RL as mentioned below

B. Concerns Regarding Algorithm Novelty

Regarding the algorithm on the learner side, we believe that "making things usable is more important than making things fancy" in terms of overall system operations. Our tailored RL model aims to solve two main problems:

• (A) Instabilities from distributed online learning.
• (B) Efficient utilization of experiences collected from different time spans across workers.

We address (A) using ReTrace and (B) using DPER , whose individual effects are validated in our ablation studies in Section 6.6 and Figure 7(b).

Beyond the design of DPER and ReTrace, our work builds upon the foundation of Generalized Advantage Estimation (GAE) and DigiRL [3] to address the challenges inherent in distributed, asynchronous reinforcement learning environments with communication delays through two key innovations. See the following details:

Foundation and Enhancement of GAE:

• Vanilla GAE estimates advantages using TD errors across multiple timesteps: $A^{\text{GAE}}(s_t, a_t) = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$

• The difference compared with GAE is the use of value estimation to estimate advantage. Firstly, we introduces trajectory-level value estimation to label trajs, i.e., a trajectory-level value estimator $V_{\text{traj}}$ (parameterized by Network $\theta$). where $V_{\text{traj}}$ is trained to capture long-term dependencies and trained with MLE loss:

$\arg\mathop{\min}\limits_{\theta} \mathcal{L}(V_{\text{traj}}) = -\mathbb{E}\_\nu [ r(s_H, a_H) \log V_{\text{traj}}(s_H, a_H) + (1-r(s_H, a_H)) \log(1-V_{\text{traj}}(s_H, a_H)) ]$

Trained trajectory critic serves dual purposes: labeling rewards and filtering the replay buffer to retain only high-value trajectories.

Given that our primary signals are extended and delayed rewards, we then train a Value-network $V(s_t)$ to estimate expected cumulative rewards directly from states rather than learning $Q$ values via trajs' success or failure signals from MC-Return. The advantage function is finally computed using the one-step formulation $A(s_t,a_t) = r(s_t,a_t) + \gamma V(s_{t+1}) - V(s_t)$, where $r(s_t,a_t)$ includes some hard-coded intermediate rewards/penalties.

It's worth mentioning that in our practice, we have off-policy corrections using ReTrace upon the value function, please see the Section 5.1-5.2 in the revised version.

Policy Optimization with Robust Regularization:

Once we have obtained advantage function $A(s_t,a_t)$ and we can train the actor based on the policy loss

• We extend Digirl's advantage-weighted objective: $\mathcal{L}\_{\text{AWR}} = -\mathbb{E}\_{\nu}[\log \pi(a|s) \cdot \exp(A(s,a)/\beta)]$ to a more robust formulation with importance sampling and regularization: $\mathcal{L} = -\mathbb{E}\_{\mu} [ \rho_t A(s_t, a_t) \log \pi(a_t | s_t) ] - \beta \mathbb{E}\_{\mu} [ \mathbb{H}(\pi) ] + \lambda \mathbb{E}\_{\mu} [ \mathcal{P}_{\text{invalid}} ]$

This formulation improves upon both GAE and [3] by:

• Incorporating importance sampling ($\rho_t$) for stable off-policy learning
• Adding entropy regularization and invalid action penalties for robust exploration

Since we did not position our work as a novel algorithm-centric paper, we did not dedicate extensive space to this explicit policy learning part and only mentioned the core designs. We have added these details to the revised version to enhance clarity.

Dear Reviewer fxPf,

Thank you sincerely for your valuable feedback on our paper.

As the discussion period comes to a close, we hope our additional experiments and thorough clarifications have satisfactorily addressed your concerns. We have dedicated significant effort to implement all your suggestions and provide an in-depth analysis.

We sincerely invite you to consider raising your scores if our responses have effectively addressed your concerns. Please let us know if there's anything further you'd like us to address—we are more than happy to continue the discussion. Your perspective is highly important to us, and we look forward to your response.

We sincerely appreciate the reviewers' valuable feedback and answer as below:

A. Main Weaknesses and Concerns

1. Hyperparameter Sensitivity and Determination

While our approach involves multiple hyperparameters, we effectively manage them through domain heuristics to constrain the parameter space and search. Detailed hyperparameter configurations and tuning procedures are documented in Appendix A.4.3.

2. Expanding to a Broader Range of Mobile Applications

We conducted additional experiments on "GoogleApps" and "Install" tasks from AiTW datasets, not included in the training set, to assess generalization. Results in Appendix A.6.2 demonstrate that DistRL maintains high performance and effectively generalizes to new tasks. However, achieving high generalization remains challenging when facing substantial variations across different apps. We are also exploring more open-world benchmarks, and as future work, we plan to expand our exploration to a wider range of mobile applications.

3. Credibility of our evaluator

To validate Gemini's reliability (According to our rewarding design, we only need to justify whether the task is successful or not), we conducted supplementary experiments comparing its evaluations with human assessments, i.e., we tested correlation between our autonomous evaluator and human judgements agents on whole General subsets from AiTW:

X is used to represent evaluation(rewarding) via Last X images

The analysis showed a high correlation with discrepancies in less than 2% of cases, demonstrating Gemini as a reliable proxy that reduces tons of manual labeling costs. Since rewarding from external evaluator is very close to the human labelling, we didn’t introduce other rewarding mechanism in our current research. But I believe using dense rewards will be a good future direction. (More analysis can be found in my responses to Reviewer w8MY)

B. Answers to Questions

Q1. Results from Other AiTW tasks and Comparison with AndroidWorld

We clarify that we used the same task set as prior work and focused our testing on AiTW tasks to enable direct comparisons with existing baselines, e.g., DigiRL [3]. While other AiTW tasks would indeed be broader, they are not suitable for scientific projects (e.g. tasks that involve accounts) or have a very slow response time (e.g. App installation). However, we have also performed experiments on the "INSTALL" and "GOOGLEAPPS" tasks from AiTW to validate generalization abilities. The results, which are included in Appendix A.6.2, Figure 11, indicate that DistRL performs well on these additional tasks. (We didn’t exclusively train our agents on these tasks but treated them as generalization test sets)

Q2. Handling Significant Changes

DistRL’s design accommodates different device types, OS versions, UI changes, and app updates by using ADB and emulators for human-like interactions and system resets. The main challenge is the compatibility of AVD and app versions. For major UI overhauls or new devices, additional training or fine-tuning may be required. However, DistRL’s architecture supports scalable data collection and training across diverse environments, enhancing its robustness in real-world scenarios. We validate this robustness using our dedicated testing infrastructure, including proprietary OS environments and hardware.

Q3.Computational Overhead and Empirical Performance trade-off

In our evaluation of the computational overhead associated with using more environments (workers) and implementing DPER, we found the overhead to be minimal compared to the performance gains achieved. Specifically, we conducted some supplementary experiments as below: Evaluating computational overhead with 96 vCPUs and 32 emulators, DPER introduced ~7% training overhead compared to without DPER, while improving the average success rate from 5.3% to 13.3% (compared with baselines [3]), which is significant in this context. Optimization techniques like caching, distributed sampling, and priority updates every 5 steps minimized overhead. Communication overhead among scaling workers and learners is managed efficiently, with plans to utilize high-speed data transfer technologies like InfiniBand or RoCE.

Q4. Adapting the Framework for Other OS

To extend DistRL to iOS, trajectory input and environment resets would utilize Apple’s tools (e.g., XCUITest, simctl) instead of Android-specific tools. The core components are platform-agnostic, and with necessary adjustments, DistRL can support iOS environments. However, Android is still the first choice now due to the developer-friendly environment.