CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs

We appreciate your insightful and valuable comments. We provide reponses to all your concerns as follows.

Q1: Adaptability to image-based setting.

A1: We first clarify that we initially chose to build CORE on XML-based UI representations because, prior to 2025, foundation multimodal LLMs lacked sufficient reliability for accurate UI element localization. In constrat, XML provided precise bounding box information and a structured hierarchy that is essential for accurate decision-making. We would also like to clarify that, in practice, XML extraction works on most pages across mainstream apps.

Of course, we agree with the reviewer that visual-based agents are rapidly advancing, and we believe XML and screenshots each have complementary advantages. Screenshots offer richer visual context, while XML encodes semantic metadata (e.g., content-description), reflecting developer intent.

In fact, our CORE framework is modality-agnostic and can be conveniently adapted to incorporate screenshots. This is because our key contribution lies in proposing a general workflow designed to address a new problem: reducing unnecessary UI exposure to the cloud LLM while preserving decision quality. All the key modules of layout-aware block partitioning, co-planning, and co-decision-making are all modality-agnostic. Each module can operate on XML-only, screenshot-only, or combined inputs. For example, the block partitioning module can use either an XML-based strategy or a visual segmentation approach. Similarly, the local and cloud LLMs can be replaced with multimodal LLMs. The rest of the collaborative pipeline remains unchanged. This resonates with the reviewer's suggestion regarding visual-based segmentation or ranking, but it may need some engineering efforts.

To extend CORE to multimodal input, we have adopted a straightforward masking strategy: CORE’s selective UI reduction enables us to precisely identify which UI elements should be masked. We capture screenshots of the UI and apply gray masks to the bounding boxes of elements that CORE removes, generating a masked image. This masked screenshot is then passed to a cloud multimodal LLM (e.g., GPT-4o in our evaluation) for decision-making, alongside the textual UI element descriptions used in the original XML-based setting. The GPT-4o baseline receives the full screenshot, while our CORE uses the masked one.

We show the evaluation results on DroidTask and AndroidLab as follows.

Results on DroidTask:

Results on AndroidLab:

These results show that CORE remains effective with multimodal input: it achieves success rates close to the GPT-4o baseline while reducing UI exposure by 56.84% and 37.51% on DroidTask and AndroidLab, respectively.

Q2: Case study to understand the exact form of the sub-tasks.

A2: We have included several practical case studies in Appendix E (Figures 1–5) of the supplementary material, illustrating multi-step, cross-screen trajectories under our CORE. We agree with the reviewer that adding a dedicated case study section in the main text would greatly improve clarity. In the revision, we will include a step-by-step case study that illustrates the sub-task generation and selection process within the CORE framework. For example, for the trace of task “Change theme color to light”, the sub-tasks generated at each co-planning step are:

• “Look for the settings or theme selection option.”
• “Open Settings”
• “Navigate to the theme color setting.”
• “Select the light theme option.”
• “Save the change.”

These sub-tasks are generated by the cloud LLM (without direct UI access) using only the overall task goal, interaction history, and candidate sub-tasks proposed by the local LLM from different UI regions.

Q3: Overall system cost analysis.

A3: We have presented the cost in Section 4.2 (Cost Analysis) and Figure 4 (Left) of the submission, which includes a full breakdown of both cloud and local latency. To provide a clearer view, we present it in the following tabular format, as shown below:

System Cost on DroidTask:

System Cost on AndroidLab:

Regarding the reviewer’s valuable suggestion to optimize the “Rank blocks by current sub-task” step using a dedicated ranking model, we agree this is a promising direction. However, our preliminary attempts with either using smaller general-purpose LLMs (3B) or ranking UI blocks based on semantic similarity to the current sub-task did not yield satisfactory ranking performance. We consider that a dedicated lightweight ranking model would require task-specific training, which should be an important direction for future work.

Q4: Keep the local LLM consistent in Figure 5 (right).

A4: We apologize for the confusion. In Figure 5, we used different local LLMs because we prioritized the best-performing setup for each dataset in terms of task success rate. The optimal local LLM varies on DroidTask and AndroidLab datasets. In the revision, we will either keep the local LLM across both datasets consistent in the figure or clearly explain in the figure caption why different local LLMs were chosen.

Thanks so much for your thoughtful comments. We've addressed all your concerns in detail below.

Q1: Preventing the exposure of sensitive information and practical significance of the Reduction Rate metric.

A1: We agree with the reviewer that the "sufficient UI content" necessary for task completion may inevitably include private information. From the outset, our design does not claim to eliminate all sensitive exposure, but rather aims to reduce the scope and amount of potential privacy leakage. By selectively uploading only the most relevant UI blocks, our approach reduces unnecessary privacy exposure, thereby lowering the probability of transmitting task-irrelevant sensitive content to the cloud.

We have also followed the suggestion to conduct a quantitative privacy analysis on the sensitive information omitted by CORE, specifically, comparing the sensitive elements uploaded to the cloud by the GPT-4o baseline and our method. We categorize sensitive data into 8 categories: (1) Identity & Account(e.g., username, profile); (2) Location & Schedule(e.g., home address, calendar events); (3) Contacts & Communication(e.g., contact information, messages, call logs); (4) Media & Files(e.g., file name, file content); (5) Device & Usage Info(e.g., device ID, storage); (6) Behavior & Preferences(e.g., interests, history, custom settings); (7) Finance & Security(e.g., payments, passwords, transactions); and (8) Other Sensitive Information. For each step in a task, we have analyzed all uploaded UI elements using Qwen2.5-max to identify sensitive content and assign each sensitive UI element to one of the above categories.

Results on DroidTask:

Results on AndroidLab:

The results show that our CORE significantly reduces sensitive UI exposure by 70.49% and 38.84% on DroidTask and AndroidLab, respectively. These reductions align with the overall element reduction rates of 55.60% and 34.96% reported in the submission, indicating that reducing overall UI exposure effectively lowers the risk of privacy leakage.

We have further manually inspected and understood where the reductions come from. The key findings are that most sensitive data fell into the Location & Schedule and Contacts & Communication categories due to apps like Calendar, Contacts, and Messenger. We show some detailed cases of our CORE as follows:

• In a Calendar task (creating a new event), only the context relevant to the new event is uploaded, omitting previously scheduled events on the screen that reveal the user’s calendar.
• In a Contacts task (updating a phone number), only the relevant phone number field is uploaded, while fields like email and birthday are excluded.
• In a Messenger task (sending a message), only the current message block is uploaded to the cloud; prior chat history is not.

The above results have demonstrated that our CORE reduces unnecessary sensitive UI exposure, indeed increasing user comfort and trust in the mobile agent.

Q2: Technical novelty of CORE, compared to previous multi-agent work [1,2].

A2: We first clarify that prior work explored multi-agent framework design for the objective of improving task accuracy. In contrast, our CORE is developed for a different new objective: minimizing unnecessary UI exposure to the cloud LLM without compromising decision quality.

The difference on the design objective also leads to the key difference on the design settings. Prior work typically leveraged strong cloud LLMs with different prompts/roles [1, 2] and tools (e.g. retrieving external knowledge [2]), where they all can access the full UI. In contrast, our CORE coordinates between a cloud strong LLM and a local weak LLM. In particular, the powerful cloud LLM has strong reasoning ability but cannot access the full UI, whereas the weak local LLM has access to the full UI but has limited reasoning capacity. Since the cloud LLM cannot access the full UI while also needs to identify which inputs are necessary for planning and decision-making, this strongly motivates the collaboration between the cloud strong LLM and the local weak LLM for co-planning and co-decision-making.

The above new settings finally lead to the following design novelty in our CORE pipeline:

• Our co-planning stage and cloud–local LLMs collaboration in it differ from typical task decomposition designs. Here, the local weak LLM does not attempt to directly determine the correct sub-task. Its limited reasoning ability makes this unreliable. Instead, it generates UI-region-based sub-task candidates without concrete UI content, which are then passed to the cloud strong LLM. The cloud strong LLM, leveraging its stronger reasoning ability and task history (but without seeing the full UI), selects and refines the most appropriate sub-task.
• In the co-decision-making stage, unlike prior work that focuses solely on decision accuracy, we aim to balance decision quality and UI reduction. The local LLM decides which UI blocks to upload, as the cloud LLM cannot identify them without direct UI access, but needs to make decision. The validated sub-task from co-planning plays a key role in helping the local LLM rank and select necessary blocks. The cloud LLM makes decisions using incomplete but sufficient UI blocks provided by the local LLM.

[1] Wang J, Xu H, Jia H, et al. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration[J]. arXiv preprint arXiv:2406.01014, 2024.

[2] Agashe S, Han J, Gan S, et al. Agent s: An open agentic framework that uses computers like a human[J]. arXiv preprint arXiv:2410.08164, 2024.

Q3: Evaluation on AndroidWorld.

A3: We have followed the suggestion to supplement the evaluation on AndroidWorld. We clarify that our pipeline relies on the preprocessing of XML trees extracted via uiautomator2, which differs from the default UI extraction tool used by AndroidWorld’s A11Y_FORWARDER_APP. The difference of XML tree extraction affects and requires the reconstruction of the default block partitioning module in our CORE, which leverages the XML layout to group semantically related elements and faciliates better block filtering. Due to time limitation, we directly use the UI elements provided by AndroidWorld and evenly divide them into three blocks, while the subsequent two stages of CORE remain unchanged. The evaluation results of our CORE with the naive even block partitioning are shown as follows.

We can observe that our CORE framework achieves 45.12% of UI reduction, and the success rate decreased from 35.34% to 24.13%. The drop in success rate is partly attributable to the naive even block partitioning strategy and the inclusion of some very long-horizon and question-answering tasks in AndroidWorld. As validated in our ablation study in Section 4.3, replacing the XML-layout-informed partitioning with naive even partitioning can degrade task success rate (e.g., drop by 6.99% on DroidTask), but it allows us to test the feasibility of our overall framework. We will include more comprehensive results on AndroidWorld in the revision.

Q4: Case study.

A4: We have included several practical case studies in Appendix E (Figures 1–5) of the supplementary material, illustrating multi-step, cross-screen trajectories under our CORE. We agree with the reviewer that adding a dedicated case study section in the main text would greatly improve clarity. In the revision, we will highlight some typical cases to qualitatively demonstrate our UI exposure reduction and the corresponding privacy gains, some of which are briefly illustrated in our response A1 to your comment Q1.

Thanks for your thorough comments. We have provided detailed responses to your key concerns as follows.

Q1: Quantitative privacy gains and user study for perceived comfort or trust.

A1: We have followed the suggestion to conduct a quantitative privacy analysis on the sensitive information omitted by CORE, specifically, comparing the sensitive elements uploaded to the cloud by the GPT-4o baseline and our method. We categorize sensitive data into 8 categories: (1) Identity & Account(e.g., username, profile); (2) Location & Schedule(e.g., home address, calendar events); (3) Contacts & Communication(e.g., contact information, messages, call logs); (4) Media & Files(e.g., file name, file content); (5) Device & Usage Info(e.g., device ID, storage); (6) Behavior & Preferences(e.g., interests, history, custom settings); (7) Finance & Security(e.g., payments, passwords, transactions); and (8) Other Sensitive Information. For each step in a task, we have analyzed all uploaded UI elements using Qwen2.5-max to identify sensitive content and assign each sensitive UI element to one of the above categories.

Results on DroidTask:

Results on AndroidLab:

The results show that our CORE significantly reduces sensitive UI exposure by 70.49% and 38.84% on DroidTask and AndroidLab, respectively. These reductions align with the overall element reduction rates of 55.60% and 34.96% reported in the submission, indicating that reducing overall UI exposure effectively lowers the risk of privacy leakage.

We have further manually inspected and understood where the reductions come from. The key findings are that most sensitive data fell into the Location & Schedule and Contacts & Communication categories due to apps like Calendar, Contacts, and Messenger. We show some detailed cases of our CORE as follows:

• In a Calendar task (creating a new event), only the context relevant to the new event is uploaded, omitting previously scheduled events on the screen that reveal the user’s calendar.
• In a Contacts task (updating a phone number), only the relevant phone number field is uploaded, while fields like email and birthday are excluded.
• In a Messenger task (sending a message), only the current message block is uploaded to the cloud; prior chat history is not.

The above results have demonstrated that our CORE reduces unnecessary sensitive UI exposure, indeed increasing user comfort and trust in the mobile agent.

Q2: Technical novelty of CORE compared with prior work [1,2,3].

A2: We first clarify that prior work explored two-stage designs for the objective of improving task accuracy. In contrast, our CORE is developed for a different new objective: minimizing unnecessary UI exposure to the cloud LLM without compromising decision quality.

The difference on the design objective also leads to the key difference on the design settings. Prior work typically leveraged strong cloud LLMs with different prompts/roles [1], tools [3], or designed specialized models [2], where they all can access the full UI. In contrast, our CORE coordinates between a cloud strong LLM and a local weak LLM. In particular, the powerful cloud LLM has strong reasoning ability but cannot access the full UI, whereas the weak local LLM has access to the full UI but has limited reasoning capacity. Since the cloud LLM cannot access the full UI while also needs to identify which inputs are necessary for planning and decision-making, this strongly motivates the collaboration between the cloud strong LLM and the local weak LLM for co-planning and co decision-making.

The above new settings finally lead to the following design novelty in our CORE pipeline:

• Our co-planning stage and cloud–local LLMs collaboration in it differ from typical task decomposition designs. Here, the local weak LLM does not attempt to directly determine the correct sub-task. Its limited reasoning ability makes this unreliable. Instead, it generates UI region-based sub-task candidates without concrete UI content, which are then passed to the cloud strong LLM. The cloud strong LLM, leveraging its stronger reasoning ability and task history (but without seeing the full UI), selects and refines the most appropriate sub-task.
• In the co-decision-making stage, unlike prior work that focuses solely on decision accuracy, we aim to balance decision quality and UI reduction. The local LLM decides which UI blocks to upload, as the cloud LLM cannot identify them without direct UI access, but needs to make decision. The validated sub-task from co-planning plays a key role in helping the local LLM rank and select necessary blocks. The cloud LLM makes decisions using incomplete but sufficient UI blocks provided by the local LLM.

[1] Dai G, Jiang S, Cao T, et al. Advancing mobile gui agents: A verifier-driven approach to practical deployment[J]. arXiv preprint arXiv:2503.15937, 2025.

[2] Christianos F, Papoudakis G, Coste T, et al. Lightweight neural app control[J]. arXiv preprint arXiv:2410.17883, 2024.

[3] Mei K, Zhu X, Xu W, et al. Aios: Llm agent operating system[J]. arXiv preprint arXiv:2403.16971, 2024.

We sincerely thank the reviewer's insightful and valuable comments. We have provided responses to address all your concerns as follows.

Q1: Performance on tasks requiring multiple steps across multiple screens or highly dynamic UIs, like social media apps.

A1: Regarding multi-step, multi-screen tasks, we want to clarify that both benchmark datasets used in our evaluation (DroidTask and AndroidLab) are composed of multi-step, multi-screen tasks:

• In DroidTask, according to Figure 6(a) in its paper [1], human trajectories span an average of 5 screens, with some tasks involving over 10 screens.
• In AndroidLab, as shown in Figure 5(f) of its paper [2], tasks are even more complex, requiring on average 7 screen transitions, with some tasks over 10 screens.

We have also included several practical case studies in Appendix E (Figures 1–5) of the supplementary material, illustrating multi-step, cross-screen trajectories under our CORE.

Regarding social media apps with highly dynamic UIs, we have followed your suggestion to add experiments on a subset of the LlamaTouch dataset [3], comprising tasks from common social media apps. We tested 64 tasks across Instagram, X (Twitter), Reddit, and Pinterest, with the following results:

Evaluation results show that our CORE completed only 6 fewer tasks than the full-UI GPT-4o baseline, while reducing UI exposure by 48.94%, revealing effectiveness on social media apps with highly dynamic UIs.

[1] Wen, Hao, et al. "Autodroid: Llm-powered task automation in android." Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 2024.

[2] Xu, Yifan, et al. "Androidlab: Training and systematic benchmarking of android autonomous agents." arXiv preprint arXiv:2410.24024 (2024).

[3] Zhang, Li, et al. "Llamatouch: A faithful and scalable testbed for mobile ui task automation." Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 2024.

Q2: Local LLM’s overhead and resource demands on low-end devices.

A2: We have followed your suggestion to deploy a quantized Qwen2.5-7B-Instruct that has been used in our evaluation on a Xiaomi smartphone using Alibaba’s mobile inference engine MNN. We select all 5 tasks in Applauncher app from DroidTask dataset for overhead testing. The smartphone's hardware configurations, the inference latency, the CPU usage, and the memory usage per task are shown below.

We can observe that the inference latency of on-device LLM is roughly 140s per task, the CPU roughly occupies 8 cores, and the memory usage is below 7GB.

We further have adopted more universal LLM engine llama.cpp to deploy all three local LLMs used in our evaluation for overhead testing, including Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-2-9B-Instruct. We find that the latency of Qwen2.5-7B-Instruct using llama.cpp is about 5× higher than MNN, and the CPU and memory usage of llama.cpp are comparable to MNN. In addition, the overhead increases slightly with the model size increasing from 7B to 9B.

We finally want to clarify that efficient LLM inference on mobile devices is still a great challenge in existing mobile agent work, primarily due to limited hardware resources. As a result, existing work typically called cloud LLM APIs or ran LLMs on more powerful computers and servers. We initially followed this common practice using a consumer-grade RTX 4090D GPU for overhead testing with the results of Qwen2.5-7B-Instruct shown below. We can find that the inference speed on 4090D is 4.39× faster than using MNN on Xiaomi 15 Pro.

Q3: Adaptability to screenshot-based or multimodal settings.

A3: We first clarify that we initially chose to build CORE on XML-based UI representations because, prior to 2025, foundation multimodal LLMs lacked sufficient reliability for accurate UI element localization. In constrat, XML provided precise bounding box information and a structured hierarchy that is essential for accurate decision-making.

We also agree with the reviewer that multi-modality is important, as XML and screenshots each have complementary advantages. Screenshots offer richer visual context, while XML encodes semantic metadata (e.g., content-description), reflecting developer intent.

In fact, our CORE framework is modality-agnostic and can be conveniently adapted to incorporate screenshots. This is because our key contribution lies in proposing a general workflow designed to address a new problem: reducing unnecessary UI exposure to the cloud LLM while preserving decision quality. All the key modules of layout-aware block partitioning, co-planning, and co-decision-making are all modality-agnostic. Each module can operate on XML-only, screenshot-only, or combined inputs. For example, the block partitioning module can use either an XML-based strategy or a visual segmentation approach. Similarly, the local and cloud LLMs can be replaced with multimodal LLMs. The rest of the collaborative pipeline remains unchanged.

To extend CORE to multimodal input, we have adopted a straightforward masking strategy: CORE’s selective UI reduction enables us to precisely identify which UI elements should be masked. We capture screenshots of the UI and apply gray masks to the bounding boxes of elements that CORE removes, generating a masked image. This masked screenshot is then passed to a cloud multimodal LLM (e.g., GPT-4o in our evaluation) for decision-making, alongside the textual UI element descriptions used in the original XML-based setting. The GPT-4o baseline receives the full screenshot, while our CORE uses the masked one.

We show the evaluation results on DroidTask and AndroidLab as follows.

Results on DroidTask:

Results on AndroidLab:

These results show that CORE remains effective with multimodal input: it achieves success rates close to the GPT-4o baseline while reducing UI exposure by 56.84% and 37.51% on DroidTask and AndroidLab, respectively.