Cradle: Empowering Foundation Agents towards General Computer Control

We thank the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

Q1: In several parts of the paper, the authors attribute the agent’s poor performance on certain tasks to the limited OCR and visual capabilities of the underlying LLM. However, no formal evaluation of these limitations is presented in the main paper. While Appendix E5 briefly discusses GPT-4's perception limitations, it lacks a systematic analysis or quantifiable assessment.

A1: Thanks for pointing it out. Due to the limited pages in the main paper, we mainly focus on introducing GCC setting and Cradle framework and try our best to provide additional useful information in the appendix. Systematic analysis of the LMMs visual capabilities is indeed important, which is worth a separate paper for study. There are already some recent works[1] on it, whose conclusion is similar to ours that "VLMs consistently struggle with tasks that require precise spatial information and recognizing geometric primitives that overlap or are close together."

[1] Rahmanzadehgervi, Pooyan, et al. "Vision language models are blind." ACCV 2024.

Q2: The reliance on prior layout familiarity poses a limitation to the agent’s ability to generalize, in my opinion. Ideally, the agent should identify interactive elements based on general characteristics, rather than relying on previously encountered layouts, which undermines its robustness in unfamiliar software environments.

A2: Thanks for the insightful comment, which is actually not contradictory to our statement in the paper. The reliance on prior layout familiarity is exactly caused by the fact that current LMMs do not have the ability to reliably identify the interactive elements based on general characteristics. As we mentioned in the paper, take Outlook as an example, when we are writing an email, we need to click on the blank area to the right of the "To" element and then enter the email address, rather than clicking on the "To" itself. We find that it is quite difficult for GPT-4o to enter the address successfully since the blank area does not look like an interactable element. Instead, it will keep trying to click the "To" element, which in their vision looks more like an interactable button. It is a common challenge for all web/software agents and there is still significant room for improvement in LMM's UI understanding.

Q3: How do the authors propose to address the issues with impaired visual capability. In situations like the one cited below as it seems that even if the agent can recognize every object in the scene, it would still struggle due to a lack of "attention" or focus.

L359: "Additionally, indoor tasks like Search for Supplies are also challenging due to GPT4-o’s limited spatial perception, which finds it difficult to locate target objects and ends up circling aimlessly around the house. Moreover, the room contains numerous interactive items unrelated to the task, resulting in much more steps for the agent to complete the task."

A3: Recognizing objects in the wild is much more challenging than recognizing objects independently. The lack of "attention" or focus mentioned by the reviewers can still be explained by the fact that the model is not robust and generalized enough, where the performance has a significant drop in complex cases. The general solution is always improving LMMs’ visual understanding abilities, however, it may not be affordable for most researchers. Another temporal solution is to apply advanced domain-specific tools to augment the visual ability, like Grounding Dino and SAM. These tools are usually more lightwise and have better performance in their own domain compared to LMMs.

Thanks for pointing it out. Yes, that is why we call it a temporal solution. Though it cannot totally solve the problem, it indeed alleviates the issue. We also want to point out that these tools have improved quickly. For example, when we were developing Cradle, what we used were Grounding Dino and SAM. In recent days, Grounding Dino 1.5 and SAM 2 are already available. A performance gain would be expected if these tools are replaced with the latest advanced version.

Q7: The baseline comparisons in section 4.1. do not seem all that useful/informative to me. It is not clear what is being 'ablated' compared to Cradle (GPT-4o) in Table 3.

A7: Sorry for the confusion. There is a mistake in the caption of Table 3, where the ablation studies should be baseline comparisons. Table 3 is also only mentioned in the Baseline Comparison Section. We have fixed it in the latest version.

Q8: The ablations in 4.2 could be expanded to at least one non-game scenario tasks. The authors present the system as one that generalizes across multiple domains and it would be useful for readers to know what elements of the system are most helpful for enabling that generalization or indeed if non-video game domains need the same set of components as video game domains. I'd encourage the authors to run the ablations over the OSWorld benchmark or a benchmark like WebArena (which has wider adoption) that enables automatic evaluation.

A8: Thanks for the suggestion. Before the deadline of the submission, OSWorld is the only benchmark that satisfies GCC setting, which directly uses keyboard and mouse actions to interact with. Other benchmarks like WebArena only provide the interactable elements ID obtained from the internal API for the agents to interact with, which violates the GCC setting.

Running experiments on such a comprehensive benchmark is time- and budget-consuming. As mentioned in Table 6, a single run on OSWorld would cost about 500 USD and 240 hours. We hope the reviewers can understand that it is too costly to conduct ablation studies on such a benchmark.

Q9: Another ablation that I think would make this a stronger paper is a version of Cradle where the LLM doesn't have access to tools such as Grounding DINO, SAM and others. Is that they key ingredient to make systems like this effective.

A9: Thanks for pointing it out. We would like to mention that these tools are integrated within the Information Gathering module. When the module is ablated in the ablation studies, these tools are also unavailable. Only RDR2 uses Grounding DINO to help with enemy detection, where the performance will drop from 20% to 0 if without the tool in the combat task. These tools are only used to solve the rare cases that LMM can not handle. In most cases, Cradle still relies on the LMMs’ own visual abilities. Other games like Stardew Valley, Cities: Skylines and Dealer’s Life 2 do not rely on these tools.

Q10: Table 1: What are the units for "Farm Clearup (Grids Num)" and "Avg Haggling (Count)"?

A10: The unit is exactly shown in the brackets, namely the number of grids that the agent clears up and the average count of haggling per deal.

Q11: When you use the plus or minus sign, is that reflecting the range in both directions? Cities Skylines Human population is 415+-416, which would imply some player had -1 population? This table might be clearer if it indicated the min and max separately.

A11: 415±416 indicates that the mean of the population is 415 with a standard deviation of 416, which is the standard and most widely accepted expression. The population should be at least 0 without an upper limit.

Q12: Table 4: I think this should be be "Figure" 4. I do find it hard to read, what do the shaded areas represent? They seem to overlap a fair amount, how should we interpret this?

A12: It is the error bar, which can be found in almost every RL and other decision-making paper. It represents the standard deviation of the line with the same color. The overlaps are quite common in this kind of figure, which is usually caused by two lines with close value and std. They do not have any special meaning.

We thank the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

Q1: About the presentation of the work.

A1: We deeply understand the reviewer’s confusion from the comments and sincerely appreciate the reviewers’ efforts and time to write such a long review. However, we are quite confident that the paper tells a complete story. The takeaways are apparent and easy to get, which are clearly stated in the abstract, introduction and conclusion sections.

After carefully reading the reviewers’ review more than ten times, we believe the main conflict comes from the different positions and expectations, namely the contributions that we have regarding Cradle framework in this paper, rather than the presentation, so that the reviewers may expect more presentation which is actually not the focus of the paper. It is partially caused by the reason that the reviewers are not familiar with the domain and the related works, resulting in difficulties in understanding the General Computer Control (GCC) setting. We here try our best to make some clarifications to bridge the domain gap.

1.1 GCC Setting

The primary contribution of this paper should be attributed to the GCC setting. The GCC setting provides a blueprint for how the agents can interact with any software without access to the built-in APIs, whereas almost all of the previous works heavily rely on the APIs resulting in the lack of generalizability.

This setting mainly has three advantages:

1. It could save huge efforts in designing specific wrappers for computer environments. Traditionally, researchers have to design specific observation spaces and action spaces to encapsulate the raw environments into research environments, which usually needs a comprehensive understanding of the environment and a period of hard work. Agents under GCC settings can directly be deployed into any computer environment in the wild with the unified interface.

2. It greatly extends agents’ reach to the tasks and software that have never been explored due to the lack of existing research encapsulation. Most current research environments are based on open-source environments, where they can directly obtain the internal state information and execute various actions through APIs with the designed encapsulation. The encapsulation is difficult to design in close-sourced software without access to built-in APIs. Under our GCC setting, agents can directly interact with software in the wild without the precondition of access to APIs. They can be developed into enormous commercial software, especially video games, which are precious undiscovered treasures, to learn and evaluate various abilities.

3. Under GCC setting, all the data is collected in the same format and granularity since agents interact with various environments in a unified way, which enables comprehensive training for general improvement towards general intelligence.

We thank the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

Q1: While the evaluations are extensive, the number of runs and human participants remains relatively small to achieve robust statistical significance. Given the complexity and resources required for such evaluations, the authors might consider acknowledging this limitation explicitly in the paper.

A1: Thanks for pointing it out and we deeply appreciate the reviewer's understanding of the complexity of the evaluations. We totally agree that the limited number of human participants is one limitation of this paper. We have updated it in the latest version.

Q2: Could the authors clarify the specific visual prompting techniques employed, particularly the extent of manual curation involved? Greater transparency here would help assess the generalizability of the framework.

A2: Thanks for pointing it out. Due to the limitation of pages, the visual prompting techniques we used were introduced in Figure 21 (Page 36) and Figure 29 (Page 46) in the appendix. The main idea of them is to use colorful bands to improve LMMs’ sense of direction and draw coordinates and axis on the screenshots to improve LLMs’ visual understanding and pixel-level control.

Q3: Were task-specific prompts manually written, or were they generated—at least partially—by LMMs? If generated, what iterative methods were employed to refine these prompts?

A3: When we start to work on a new software/game, we will first ask LMMs to analyze task/domain-specific stuff that each module needs to pay attention to, which forms the initialization of the domain knowledge into the prompt template. Then we will iteratively modify the prompts according to the agent's performance manually with the help of LMMs, to make sure the prompts align with LMMs conventions and are free of ambiguity. How to let LMMs generate the prompts automatically and iteratively is an insteresing and promising open question.

Q4: Can the authors elaborate on the design rationale for each module, discussing why each module was considered necessary and the factors underlying their inclusion, rather than directly presenting each module?

A4: As we introduced in the paper, the philosophy behind the design of the modules is a straightforward problem-solving mindset, whose reasoning process is analogous to "reflect on the past, summarize the present, and plan for the future".

Information Gathering and Self-Reflection are used to analyze the performance and outcomes of the preceding action's execution.

Task Inference and Memory summarize the current situation and analyze the current progress of the task based on the reflection results.

Skill Curation and Action Planning generate and select skills to be executed in the next actions, according to the current situation and the progress of the task.

We are glad to see the framework motivated by the intuitive but powerful problem-solving mindset works well across domains.

Q5: For novel tasks, how much time and expertise are required to develop prompts and input videos for CRADLE to generalize effectively? An exploration of these requirements could provide a clearer picture of the framework's adaptability.

A5: One example we can provide is that a group of amateurs managed to develop Cradle into the Black Myth: Wukong and defeated some Bosses within 2 weeks. Note that this game is an extremely challenging action game and it was released 3 months ago, which is later than the knowledge cutoff date of GPT-4o.

We thank the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

Q1: Overall, this is a good systems paper with limited novelty and contribution in the representation learning space.

A1: We gratefully appreciate the reviewer’s acknowledgment of our paper on the system/application side. We want to argue that our novelty and contribution are still within the scope of representation learning, which is actually a large and general concept in machine learning including both proposing new methods for learning better representation and also the application of representation learning method.

First of all, according to according to ICLR2025 Call for Papers (https://iclr.cc/Conferences/2025/CallForPapers), "We consider a broad range of subject areas including feature learning, metric learning, compositional modeling, structured prediction, reinforcement learning, uncertainty quantification and issues regarding large-scale learning and non-convex optimization, as well as applications in vision, audio, speech, language, music, robotics, games, healthcare, biology, sustainability, economics, ethical considerations in ML, and others."

Representation learning is just one subject area in the listed 20 subject areas. Other subject areas include "datasets and benchmarks", "infrastructure, software libraries, hardware, etc" and applications to various domains. The novelty of papers in these subject areas usually does not come from proposing a novel method to learn better representation.

Our selected area of this paper is applications to robotics, autonomy, planning, focusing on the application of representation learning. As a reference, MetaGPT[1], accepted by ICLR2024 as oral presentation, which proposes a code-generation framework for LLM-based multi-agent collaborations without training, was also submitted to this primary area. They also focus on building LLM "systems" instead of "representation learning", which is exactly the same as our paper. Another famous paper, ReAct[2], accepted by ICLR2023 as notable-top-5%, which is one of the most popular agent frameworks, was also submitted to the area of applications.

We hope the reviewer can re-evaluate the novelty and contribution of our paper based on the above information since the reviewer has already acknowledged the novelty and contribution of our paper required by our selected area.

[1] Hong, Sirui, et al. "Metagpt: Meta programming for multi-agent collaborative framework." ICLR 2024.
[2] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023.

Q2: The authors should start the paper with clear input & output examples, where Fig 15 in the appendix is cumbersome to fit in front. A higher level pipeline [input, output, objective] should have been shown and then specific details on sub-objectives and I/O.

A2: We agree that the paper should start with the input & output example, which is exactly how we organize the paper. At the beginning of the framework section, we use Figure 3 to show the input and output of each module to provide a high-level understanding to the readers. Then in Figure 5, we provide 3 complete examples in different games to show how Cradle works during each step with real input and output examples. We appreciate the reviewers’ thorough reading of the appendix and their insightful observations regarding Figure 15. However, Figure 15 does not show the complete workflow of each module, which is used as a case study of how the reflection module works. We believe Figure 3 and 5 are sufficient to demonstrate the input & output and provide a precise understanding of our framework.

As for the technical challenges and specific details, we argue that Cradle is a general framework rather than a specific method, where each module can have different env-specific implementations to deal with env-specific issues. As a general framework, Cradle does not aim to focus on one or two software and their corresponding implementations. The framework section in the main paper should focus on providing a clear explanation of the framework instead of any specific implementations. And we also agree that the technical details are very important, so we tried our best to provide all the details we can provide in the appendix for the readers.

Q8: How would have Deep RL fared under this scenario with game related rewards?

A8: Thanks for pointing it out. This is exactly one of the challenges of GCC setting. In these closed-source environments, it is extremely difficult to design reward functions as encapsulations of the environments since all the users receive are screenshots/videos from the monitor without any internal information. There are no available easy-to-obtain reward signals for RL agents to learn.

Q9: Is there any LMM query prompting level innovation like chain of thought incorporation

A9: It is not clear to us what is the LMM query prompting level innovation. To our understanding, CoT is a method that unlocks the potential reasoning ability of LMMs. The following works like ReAct[1] and Reflextion[2] also attempt to build reasonable pipelines to increase the reasoning ability. Cradle also serves a following work of them. If the reviewers acknowledge the innovation made by ReAct and Reflextion, then they should also acknowledge the innovation made by Cradle.

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023.
[2] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." NeurIPS 2023.

Q10: The metrics of evaluation vary per dataset / game / software category. How can this be generalized to reduce human input?

A10: We would like to note that due to the rich content in games and software, it is difficult to use only one or a few metrics for them. Different games may evaluate agent’s different abilities, just like that each industry has its own metrics for measurement, e.g., sellers use the profits, runners use the time and students use the scores. For the goal-based tasks, a common metric would be the success rate. For the creative tasks, there are usually various metrics for them.

Q11: What mathematics and philosophy is behind the fact that this chain of 6 modules will work perfectly, and not any other combination?

A11: As we introduce in the section 3.3, Cradle’s design represents a problem-solving mindset, whose reasoning process is analogous to "reflect on the past, summarize the present, and plan for the future".

Information Gathering and Self-Reflection are used to analyze the performance and outcomes of the preceding action's execution.

Task Inference and Episodic Memory summarize the current situation and analyze the current progress of the task based on the reflection results.

Skill Curation and Action Planning generate and select skills to be executed in the next actions, according to the current situation and the progress of the task.

We will not say this framework works perfectly as if we add more auxiliary modules, the performance can still be improved, which is a trade-off between the performance and time/budget-cost. However, as shown in the ablation studies, removing any of them will result in a significant performance drop.

Q12: Were there any robust human user studies done domain and demography nostic?

A12: We are sorry that we may not fully understand the meaning of this question. It would be very helpful if the reviewers can re-clarify this question.

Q13: I was wondering how will it work for CAPTCHA task?

A13: For the CAPTCHA task, it mainly depends on the LMMs’ own ability, especially their visual ability. Due to the limited visual understanding mentioned in the paper, we do not expect the current LMMs will generally have a good performance across various CAPTCHA tasks without any specific tools.

Q14: About ethics issue. The paper, webpage and product offering is available on the internet thereby revealing author and organization details.

A14: We would like to remind the reviewers that it is not an ethics-related issue. And according to the ICLR policy, "papers that have appeared on non-peer reviewed websites (like arXiv) or that have been presented at workshops (i.e., venues that do not have publication proceedings) do not violate the policy."

Dear Reviewer w2hJ,

We sincerely appreciate your insightful comments and the acknowledgment of our paper from the system/application side. As the rebuttal deadline approaches, we would like to know whether there are any additional concerns we can address. We will try our best to provide further clarification.

GCC is a promising but extremely challenging setting where most current works try to avoid it resulting in limited generalizability. Cradle manages to show the effectiveness of GCC setting, which we hope can attract more researchers to engage in this exciting domain to achieve the general computer agents. And Cradle also serves as the first open-sourced framework for the researchers to start with. None of the above-mentioned will be possible without your support.

We hope our provided response during the rebuttal addresses the reviewers' concerns and the reviewers can re-evaluate our paper. We thank the reviewers once again for their efforts.

Best regards,
Authors of Paper 6747

Dear Reviewer w2hJ,

We want to again express our gratitude to your efforts and insightful comments. As it is the last day of rebuttal for reviewers to reply, we hope our provided responses have addressed all the concerns. Our rebuttal can be briefly summarized as following three points:

1. Application is within the acceptable scope of ICLR, which does not necessarily require novelty in learning better representation.

2. Our framework has been described in Figures 3 and 5, which clearly shows the input and output of each module and provides a precise understanding of our framework. Cradle is a general framework, instead of a specific method/implementation, which cannot include too many implementation details in the framework section. We added a “General Implementation” subsection to the Empirical Studies Section on page 6 to provide more details.

3. We did not claim contribution from any independent module, instead, the novelty comes from the whole Cradle framework as an integration. As a general framework, advanced methods can be seamlessly integrated into the corresponding module and therefore enhance the ability of Cradle.

GCC is a promising but extremely challenging setting where most current works are not applicable in this setting, which limits the generalizability of these works. Cradle manages to show the effectiveness of GCC setting, which we hope can attract more researchers to engage in this exciting domain to achieve the general computer agents. And Cradle also serves as the first open-sourced framework for the researchers to start with. None of the above-mentioned will be possible without your support.

We thank the reviewers once again for your efforts and hope reviewers can re-evaluate our paper.

Best regards,
Authors of Paper 6747