Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API

Thanks for your affirmation of our motivation, ablation study, achieved benefits and the importance of our tackled problem. We will strive to address your concerns in the following.

Q1: Is it a bit over-claimed to state that the framework does not require any additional information?
A1: Actually, it's not. This statement refers to that our framework does not require any additional information during inference. We still require the ground truth bounding boxes of UI elements as the supervisions for training. In contrast, previous work require such information in both training and inference stages since they take this information as one of the model inputs, thus limiting the practicability especially when the additional information is not available or noisy. Thanks for your question, we will further clarify this in our revision.

Q2: Does the low performance of traditional grounding methods (such as Grounding DINO [1] and GLIP [2]) on our targeted UI tasks result from them being simply under-tuned?
A2: We believe that it's not due to this reason, but because their design purposes do not align with our targeted UI tasks. Actually, we have kept the data and training configurations for these traditional grounding methods with ours to make their comparison as fair as possible. The root of their performance gaps on our targeted UI tasks lies in their different pre-training designs. The effectiveness of these traditional methods designed for object detection can be attributed to their pre-training on large-scale object-level image-text pairs in the form of contrastive learning, wherein obeject-corresponded bounding boxes are required. Two points need to be known:

1. We experimentally find that contrastive learning based pre-training does not perform well on the images densely populated with texts, e.g., documents, screenshots, for two reasons. One is that the elements (analogous to objects) in such images are too dense, making contrastive learning extremely difficult. The other is that different elements may contain one or more the same wolds, leading to a certain degree of ambiguity in the discrimination of positive and negative samples.
2. The document data used for pre-training our model is not applicable for conducting contrastive learning, due to the lack of element-wise (object-level) text annotations. The annotations of our pre-training document data correspond to all texts on the entire page, arranged in a reading order (i.e., at the page-level), without element-wise bounding boxes provided. Details can be found in this open-source repo.

Considering these, we think Pix2Seq framework is more suitable for visual grounding on the images densely populated with texts. Thus, our contributions lie not only in applying Pix2Seq to this task, but also in addressing the limitations in the training of the Pix2Seq framework. We will provide more in-depth analysis and insights in our revision.

Q3: About pre-training traditional grounding methods on document data.
A3: This is actually not feasible because of the second point detailed in our A2 above. Please note that we have pre-trained them using our UI data. But the reaults were not satisfactory due to the first point analyzed in A2 above.

Q4: Comparison to Pix2Seq on more generic tasks.
A4: We compare our reinforced Pix2Seq model RUIG to its vanilla version on object detection. The results on MS-COCO 2012 validation set are reported in the below table. To provide experiment results promptly during this rebuttal period, we train both the vanilla Pix2Seq model and ours on our code base and maintain the model configurations consistent with those used for UI visual grounding task as presented in our paper. The results demonstrate that our proposed method improves the Pix2Seq model consistently when applied to object detection and showcase its potential on more general tasks.

Thanks for the responses. Below I leave some of inline comments responding to the answers.

Q1: What prior works in this particular domain did is not of the main point here, the point is, once you have GT boxes, the problem basically becomes a grounding problem. It's just that simple. For grounding problem, the community already well-accept that the information needed in training time is the annotated boxes, and during inference time the boxes gets predicted. The problem that you're tackling here, is, a grounding problem. There is no unclarity here.

Q2 & 3: You can simply use OCR found boxes and other html-based metadata to curate a set of web-page or document pre-training set. You're simply giving unfairness to your baselines compared against with. Though I'm not particularly concerned about this, the fact that this is not even attempted, is the problem.

Q4: Thanks for the newly reported results.

Q5: The point is missed here. When collecting data, comparing the following two: (1) asking the annotator to drag a box on the element, (2) simply asking the annotator to click on intended regions (and of course, those mouse events are recorded). (2) is obviously easier for the user and more natural as a normal web-browsing behavior. When the method stated is RL, one would expect that using these surrogate or peripheral information, the actual GT boxes are not necessarily needed. In other words, you're using more than usual information, in a web-browsing behavior. (Here, I'm not asking you to perform new experiments or collect a new dataset, it is a simple motivation flaw of this paper.)

Thank you for your reply. Some misunderstandings still exist with possibly different research backgrounds. We provide further clarifications as below.

Q1: Please note that we have positioned our proposed technique as grounding, already reflected in the paper title. Our contribution is not to change the setting of grounding, but to model the automatic execution of UI tasks as a visual grounding problem. Such modeling is more practical than prior works in the research field of UI tasks. Our statement of "no additional information is required (as the model input)" is also from this perspective.

Q2 & 3: Thank you for reminding us of the potential unfairness here. A more fair way would be to compare GLIP and Grounding DINO to the version of our model that also has not undergone document pre-training (i.e., the one in Table 1). We will replace the "RUIG (ours)" in Table 3 with the "RUIG (ours) w/o pre-train" in Table 1. Note that both GLIP and Grounding DINO have conducted contrastive learning using our curated UI data, with bounding boxes derived from the metadata. However, they are not our baseline, but the "Baseline" in Table 1 is. As two classical grounding models, they are still inferior to "RUIG (ours) w/o pre-train" when directly applied to UI tasks, due to the limitations of their object-level constrastive learning in handling the data densely populated with texts. Detailed explanations have been provided above in A2.

Q5: There is exactly a misunderstanding. We did not collect data as you mentioned in (1). For UI elements, in general, their corresponding text contents and ground-truth bounding boxes are inherently included in the metadata (e.g., HTML). This allows us to obtain the training data by generating user instructions based on the extracted text contents of UI elements and automatically associate them to their corresponding ground-truth bounding boxes, eliminating the cost of manual annotation. The (2) you mentioned, in fact, requires engineering efforts in building a simulated RL environment and need to employ human annotators to interact with this environment for recording the training data. Despite this, it is another technical route worth exploring in the future.

We appreciate your valuable suggestions, as well as your throught understanding of our problem formulation and technical contributions. We response to your questions in detail below.

Q1: How about combining Eq.2 and Eq.3 and scaling the losses for coordinate tokens by rewards?
A1: Thanks for this insightful question. We conduct an experiment for this configuration (denoted by "ScaleLoss") and report the comparison results with ours as below.

We can find this configuration delivers close performance with the baseline model, with slight improvements sometimes. It does not work better than ours. Here, we could understand this experimental observation from two aspects: 1) It does not integrate coexisting relationships among different tokens into their supervisions. 2) It can be understood as a special case of the proposed policy gradient based modelling in Eq.3, since it is actually equivalent to estimating the expectation term in Eq.3 by always taking the token with the highest probability value from the distribution $P(\cdot)$ as the Monte Carlo sampling results, with other tokens ignored. However, the token with the highest reward may not necessarily the one with the highest probability value in the current distribution $P(\cdot)$, so it is prone to under-optimization.

Q2: Add the formulation or core ideas of Monte Carlo sampling.
A2: Thanks for this valuable suggestion. The core idea of Monte Carol sampling is to complete a numerical estimation based on the numerical results obtained by repeated random sampling according to a given probability distribution. We have added it in our revision and marked the corresponding part in red.

Q3: What are the adavantages of the proposed methods in terms of practical using?
A3: Sorry for this confusion. The existing methods do not model the task of localizing UI elements upon instructions as a visual grounding problem, while our this work is the first one to cast this task as a visual grounding problem. They commonly require the bounding boxes from the metadata as the model inputs, thus limiting their practical using when the metadata is not avaliable or noisy. In contrast to these model, our advantage in terms of practical using lies in that we do not rely on this information during inference since we do not take it as the model inputs.

Q4: In the comparisons to existing methods in Table 5, why not use web data?
A4: This is because other methods we compare are proposed for handling mobile UI tasks. They haven't reported their results on web data.

We are looking forward to your feedbacks on our rebuttal. Please feel free to let us know if you have new questions or suggestions. Thanks!

Thanks for your positive comments and insightful questions. We response to your questions in detail as below.

Q1: The key difference between UI tasks and general object grounding tasks [1][2][3]?
A1: Thanks for this question. The UI tasks differ from general object (visual) grounding tasks [1][2][3] in: 1) The UI screenshots are domain-specific images as they are densely populated with texts. This characteristic requires the visual grounding model for UI tasks to have sufficient OCR capability and stronger instance (instead of category) descrimination capability relative to those for general object grounding tasks. This is why we propose to adopt a Pix2Seq paradigm to align the output format of OCR (document understanding) and grounding for the utilization of pre-training. 2) Another slight difference is that the language inputs for UI tasks are user instructions which include but are not limited to element captions (with actions and other components also included), while those for general object grounding are mainly object captions.

In additon, we would like to further clarify that it is only a design choice for UI tasks to be modeled as a visual grounding problem. Besides the screenshots, the metadata containing comprehensive information of UI elements (including their names, types, bounding boxes, relationships to other elements, etc.) can also serve as the model inputs for UI element localization. In fact, most previous works cast the UI element localization task as a classification problem, and take metadata or partial information of metadata as the model inputs for learning to select the target element upon given instructions. We are the first of its kind to show the feasibility and advantages of modelling this task as a visual grounding problem.

Q2: Is the proposed reinforced learning method similar to [4]?
A2: They are similar in their high-level roles but different in the addressed problems. The similarity lies in that they both apply policy gradients based RL to enable the optimization of a global objective, but with distinct purposes in different problems. The method [4] is to enable the optimizaiton with the mAP metric over multiple objects for a vision model. Ours is designed to address the limitation issue on ignoring token-combinational semantics in the sequential decoding process for a language model.

Q3: Why do we only use half of the page to illustrate the reinforced pixel-to-sequence model?
A3: The reason for this is that we would like to benefit more readers by bringing their attentions to two takeaways of this work. The first one is for the research community on UI tasks, that is the feasibility and benefits of casting instruction-following UI task as a visual grounding problem with an improved pixel-to-sequence model. The other is about the improvement of pixel-to-sequence method itself for a broader research community. For the latter one, we think the analysis on the limitation of vanilla pixel-to-sequence model is as important as how to reinforce it in specific. A balanced solution is what we present currently.

Q4: Have the recently-proposed LLM-based multimodal models [5][6][7] been tested on UI data ?
A4: Yes, we also test them with the corresponding results reported below. Note that for this kind of models, we are not sure whether the tested Apps or Webs are seen or unseen in their training. Therefore, there are no obvious gaps between the seen and unseen settings.

Q5: (Open Question) Do we need to change the mechanism of decoded process, for example, from autoregressive to parallel decoding?
A5: This is quite an insightful open question. We are just trying to address the limitation of the commonly used autoregressive decoding based on its current mechanism by integrating the combinational semantics into its optimization process. We firmly believe in Whether we need to change the meachanism of decoded process itself? and How to change it if needed? are undoubtedly worth in-depth investigation in the future. We really hope our this work could bring more attention to these valuable research questions and inspire further exploration!

We are looking forward to your feedbacks on our rebuttal. Please feel free to let us know if you have new questions or suggestions. Thanks!

Thanks for authors' response and added experimental results, especially those of LLM-based multimodal models. I have no doubt about this submission, and I will maintain my rating.

Thanks for positive comments and valuable questions. We response to your questions in detail below.

Q: How about scaling up to the scale of the recent LMMs and comparing to them?
A: Thanks for your valuable questions for inspiring us to further investigate the potential of our proposed approach. Actually, our proposed model functions as a low-level (single-step) command executor for UI task automation. In terms of its pratical demands and design purposes, it was never anticipated to be scaled up to such an extent, in consideration to its industrial requirements (e.g., inference speed). To study its technical potential, we scale our baseline and the proposed model to the 7B scale on the current data for collecting more experimental evidences. Specifically, to obtain the results promptly, we conduct the comparison experiment between the baseline and our proposed model at this scale by employing a frozen vision encoder and fine-tuning the Vicuna-7B (an open-source improved version of LLaMA-7B) on our UI data with LoRA. They are denoted by Baseline (Vicuna-7B) and RUIG (Vicuna-7B), respectively, as below. Besides, we also test LLaVA out of box on our UI data. All results are reported in the following table.

With the constraints in time and computational resources, we have not yet tuned the baseline and our proposed model to their best during this rebuttal. Based on the current experiment results as above, we have the following observations: 1) Our proposed method consistently delivers improvements by a clear margin when scaled up to 7B. 2) LLaVA cannot work well when tested on our task right out of box, as it is designed for general object tasks. We believe that our proposed approach is still beneficial for its SFT on UI-domain data since LLaVA can be generally considered as a Pix2Seq model with an autoregressive decoding process. And our proposed approach addresses the limitation (lacking the awareness on the "combinational semantics") of current autoregressive decoding process without changing the decoding mechanism itself, as commented by Reviewer 3sqL. Therefore, we believe that, at least, it can facilitate the model optimization on a larger scale.

We are looking forward to your feedbacks on our rebuttal. Please feel free to let us know if you have new questions or suggestions. Thanks!