Thank you for your valuable suggestion.

1. Does task include test on the agent's ability?

Yes, our skill planning benchmark involve the tasks performing in Minecraft environment directly, following another Minecraft game agent work: Plan4MC. In this benchmark, our model is required to make an initial plan for a given task, adjust the plan as the agent encounters varied scenario: invalid skill names, redundant skills (already executed), successful skill execution, or failed skill execution. After skill planning, Steve-Eye execute each skill using its pre-trained low-level policies. The experiments related to skill planning are detailed in Section 4.4 of our main paper. Appendix 2 provides an in-depth look at each task. We have also conducted further experiments on long-horizon tasks, the results of which are presented in Table R4 in the response to Reviewer k9Af. These experiments demonstrates the effectiveness of our Steve-Eye in performing tasks within the Minecraft environment.

2. Questions about FK-QA Evaluation

Firstly, the QA pairs used for FK-QA evaluation are curated by ChatGPT. In order to avoid data leakage, we ensure the question prompt varies from the instruction pairs for pre-training and will not be used during pre-training.

Secondly, we have carried out experiments using a text-only benchmark with the finetuned LLaMA model. As anticipated, the finetuned LLaMA demonstrates lower performance compared to its original version, aligning with findings from prior research. However, given our objective to develop an agent proficient in games like Minecraft, we consider this performance decline acceptable. This trade-off is deemed worthwhile to achieve enhanced capabilities in our target domain for general-purpose game agents.

3. Why does LLaMA-2-7b outperform LLaMA-2-13b in Table.3?

In our observations, the LLama-2-13b model does not exhibit consistent improvements over its 7b predecessor. This trend is not isolated; similar outcomes are evident in other instruction-tuning practices based on LLaMA, including LLaVA [2]. By contrast, the LLaMA-2-70b model generally demonstrates notable enhancements when compared to both its 7b and 13b counterparts, across nearly all benchmarks.

[2] Visual Instruction Tuning

Thank you for your valuable suggestion, particularly regarding the supplementation of long-horizon tasks.

1. Long-horizon Evaluation Objectives

In fact, our work does involve long-horizon objectives in the skill planning benchmark (e.g., tasks of mining cobblestones to craft stone pickaxe'', crafting bed'' and ``harvest cooked beef''). These tasks are typically broken down into dozens of skills and may require ten thousands of steps to complete. More details of these long-horizon tasks are provided in Table 6 in Appendix.2. In addition to these tasks, we conduct experiments on 10 new long-horizon objectives. These objectives are iron-based task as detailed in Table 7 in the Appendix.2. As shown in Table.R4 , our Steve-Eye consistently exhibits promising results across these long-horizon objectives, showing its robustness in handling complex game tasks. In addition, we provide addition experiments in Table R1, R2, R3 (in reponses to ijGQ) to demonstrate that visual perception is beneficial to text-only game engines.

Table R4: 10 long-horizon iron-based tasks.

2. Is the model trained on such instruction dataset generalizable to new game? Such as GTAV?

Our Steve-Eye is not directly transferrable to other games due to each game's unique interface, especially for Minecraft known for its distinct pixel-style visuals. Nonetheless, the overall framework of Steve-Eye, including the pre-training tasks, the data collection process, and model structure, is highly adaptable and can be tailored for new games. This significantly simplifies the process of applying Steve-Eye to various open-world games. In fact, we are exploring to extend this framework to other open-ended environments, such as Virtual-Home as detailed in Appendix 4. As a simulated world, Virtual-Home demonstrates the possibility of seamlessly adapting our Steve-Eye to a range of real-world simulation games, including GTAV you mentioned.

The evaluation provided by the author for skill planning doesn't yet meet the long-horizon criteria for me. As mentioned in my comment, I am looking forward to see evaluation similar to voyager and GITM [A], such as the number of items unlocked, the extent of exploration range.

Again, as the author constructs instruction-follow dataset to train the model on those skills, it is intuitive that steve-eye can do well on them. All the evaluation is still under the generated instruction-follow dataset. I believe further evaluation is necessary to demonstrate the agent's capability to learn game-playing strategies beyond the confines of the instruction-follow dataset. Moreover, it would be valuable to see if the agent can generalize its skills to other games, given that creating instruction-follow datasets for each game is not a feasible approach.

Although my concerns remain unaddressed, I acknowledge the pioneering nature of this work in the field and will maintain my rating accordingly.

[A]. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

1. Details of Multimodal Perception Instructions

Thank you for your suggestions! Our generated responses to instructions can be divided into two segments:

• (1) Responses directly derived from processed dense captions. To foster data diversity, we instruct ChatGPT to prompt a variety of description templates as shown in Appendix 1.1. Each caption is then rephrased by randomly selecting one of these templates, ensuring a broad spectrum of expressions.
• (2) Responses to questions about the caption. We use ChatGPT to curate a set of question formats and collect QA pairs accordingly. The model is required to accurately respond based on the selected question and the visual content of the caption.

In addition, due to the space limit, we cannot provide all examples, responses, and snapshots together in the main text. We will make the paper more readable in the next version.

2. Questions about Foundational Knowledge Question Answering

Fristly, to ensure reproducibility, we will release our 1,000 QA evaluation benchmark. This evaluation process is cost-efficient, as the usage of gpt-turbo-3.5 charges only $0.0015$ per 1,000 tokens. Consequently, the entire evaluation costs less than $1$ dollar.

Secondly, it's important to acknowledge the existence of noises within our Wiki data, particularly those extracted from Wiki Tables. As you can see in https://minecraft.fandom.com/wiki, Wiki data is highly structured, which inevitably causes data noises even with careful processing. We attribute the limited performance boost to such data noises. To address this issue, we are dedicating more resources to refine the collected Wiki data for a better-quality version. Given that such an endeavor has not been previously undertaken, we believe that offering this knowledge-centric resource will significantly benefit the community.

3. Questions about Skill Prediction and Future Work

Insightful suggestion!

Firstly, regarding the use of rule-based analysis, it's not directly employed in our skill prediction evaluation but plays an important role during the pre-training of our ENV-VC task. The ENV-VC task is tailored to enable our model to interpret the current status primarily via visual cues. We acknowledge that our skill prediction lacks explicit rule-based information. However, we believe relying solely on visual cues for predictions is more in line with human's decision-making process compared to a dependency on rule-based analysis, since there won't be explicit rules in our real life.

Secondly, our initial attempt to predict moves from videos and instructions has proven to be a substantial challenge and did not yield satisfactory results before. As a result, currently our Steve-Eye assesses only the success or failure of skill execution at each step, in order that Steve-Eye can decide whether to re-plan. It does not engage in predicting low-level moves. Instead, we implement low-level moves by additionally training an RL-based policy for each involved skill. Nevertheless, we agree that directly predicting low-level moves based on visual cues could be more advantageous for creating realistic embodied agents. Thus, we are committed to exploring this direction in our future work.

It's good to know that the usage of gpt-turbo-3.5 charges only per 1,000 tokens. However, since gpt-turbo-3.5 is a commercial API, it's not sure whether it will always be available for the academic community to reproduce the evaluation. I still think an alternative evaluation which can be done offline or independent from third-party service is better for reproduction and follow-up works.

Thanks to the authors for the reply regarding the questions I raised. Overall it somehow solves my other doubts about the manuscript. I have no further questions at this point.

1. In-Depth Analysis on Skill Planning Required

Thank you for your valuable suggestions! In this refined analysis, we take a closer look at skill planning and identify several key factors contributing to the standout performance of Steve-Eye compared to other game agents like MineAgent:

• Adaptive Skill Planning Ability: MineAgent employs a multi-task policy learned through RL, which can be seen as relying on static, predefined plans to accomplish each task. In contrast, Steve-Eye adopts a different approach by decomposing a task into individual executable skills, forming new skill plans based on current, real-time conditions. This method endows Steve-Eye with exceptional adaptability in various gaming environments, enabling it to devise more effective and executable skill plans. To validate this advantage, we conduct experiments (referred to as "ours w/o decomp") in Tables R1, R2, and R3 below, which reveal a significant decline in performance without task decomposition. The results indicates that integrating task decomposition is beneficial for solving complex tasks with basic skills trained via RL.

• Additional Visual Understanding Ability: Our Steve-Eye directly leverages visual cues to assess its current status and respond accordingly. As shown in Table 5 in our main paper, visual cues empower Steve-Eye to outperforms text-only models like ``gpt-assistant'' even based on a much ligher-weight LLM. Our work also suggests a universal application of visual cues in gaming agents, moving beyond the traditional reliance on rule-based indicators for status assessment.

• More Effective In-context Reasoning Ability: As depicted in Table 3 in our main paper, Steve-Eye outperforms the vanilla Llama-2-7b by 6.4% on the foundational knowledge QA benchmark. Remarkably, it even exceeds GPT-Turbo-3.5, highlighting its acquisition of world knowledge. Therefore, Steve-Eye has enhanced adaptability to environmental changes when planning skills.

Table R1: 7 tasks involving the process of ``cutting trees to craft primary items''

Table R2: 7 tasks involving the process of ``mining cobblestones to craft advanced items''

Table R3: 10 tasks involving the process of ``interacting with mobs to harvest food and materials''

The results in Table R1, R2, R3 also validate the association between our ENV-VC, FK-QA benchmarks and the improvement of decision-making process. When the ENV-VC task is removed, and we directly utilize visual cues without further instruction tuning, there's a notable drop in performance. This clearly demonstrates the critical role of ENV-VC task in enhancing the model to intepret and utilize visual cues effectively. Similarly, the exclusion of the FK-QA task yields a performance decrease, emphasizing the essential contribution of these two pre-training tasks in enhancing our model's decision-making capabilities.