GROOT: Learning to Follow Instructions by Watching Gameplay Videos

Dear reviewer, thank you for taking the time to carefully review our paper. Thank you very much for your appreciation of the originality, quality, structure, and importance of our work, such as "The proposed idea is new, and a surprisingly simple to implement". We are also grateful for your recommendation to the ICLR community for our paper's acceptance and your belief in the potential impact of our work, as indicated by your comment "... would build upon the proposed method". We are deeply encouraged! Below, we will address the comments.

(Question 1) Details on the human participants.

Thank you for your question. We have updated these details in Appendix C.3 of the paper.

We recruited 15 students with varying degrees of Minecraft game experience, ranging from a few hours to several years, from the Minecraft project group to conduct the evaluation. They are all familiar with the basic operations of Minecraft. Each employee was asked to label 100 matches for ELO Rating or TrueSkill Rating, for a total of 1500 matches. For each employee who is required to collect or assess gameplay videos, we ask them to first read the description of each task in the Minecraft SkillForge Benchmark completely, as well as the evaluation criteria for task completion quality. Taking the task of building a snow golem as an example, the evaluation criteria are as follows: build a snow golem $\succ$ place both kinds of blocks $\succ$ place at least one kind of block $\succ$ place no block. This enables employees to quantify video quality and ensures that all employees evaluate task completion consistently. All these employees were explicitly informed that the collected data would be used for AI research.

(Question 2) Explaination of error bar in Figure 3a.

Thank you for pointing that. We found that although the ranking remained relatively stable after 1500 tournaments, there was still some variation in the ELO score around the 1500-tournament mark. Therefore, we took the ELO scores corresponding to tournament numbers 1400, 1450, and 1500, respectively, and calculated their means and standard deviations to more accurately reflect the level of the agent.

(Question 3) Explanation of error bar in Figure 5 and 6.

Thank you for raising this concern. We have updated the number of experiments to $N=10$ in the caption.

The larger variance is related to the environmental characteristics of Minecraft. Taking the task of "collect wood" as an example, we summoned the agent in a forest with complex terrain, including pools, caves, etc. We observed that when the agent accidentally fell into a pool, it took a lot of time to climb out of the water. Since the video instruction for chopping wood did not include a description of how to guide the agent to climb out of the pool, in most cases, the agent remained stuck in the pool and was thus unable to obtain any wood. If the agent did not fall into the pool, it could usually chop down a tree and obtain 4-5 blocks of wood at once, resulting in a larger variance. This phenomenon is also reflected in the STEVE-1 paper, such as its Figures 3, 4, 5, 7.

(Question 4) Environment configuration.

Thank you for raising this question. The environment settings during the inference, including whether to enable the night vision effect, are consistent across all baselines. Enabling the night vision effect is mainly to allow evaluators to clearly observe the agent's behavior and environmental changes, thus making accurate judgments. Additionally, players generally use torches to improve visibility in dark environments, meaning that most segments of the training data that involve mining in underground areas have relatively high levels of lighting. This is similar to the effect of the night vision effect. Therefore, we believe that this will not cause inconsistency between the training and inference processes.

Dear Reviewer, we sincerely appreciate your valuable time and recognition of our work in architectural design and benchmarking. We will release the entire benchmark in the future. Your comment "Results are competitive and open up for more follow-up works in this domain" has greatly encouraged us. Thank you for your questions, and please find our responses below, hoping to address any concern you may have.

(Weakness 1) Ablation on number of condition slots (learnable tokens).

Thank you for your question. We have added an ablation emperiment on numer of condition slots in Appendix D.3. We compared the performance of the model on 6 programmatic tasks with $N=1$ and $N=3$ condition slots and computed quantitative metrics for each task. As shown in the Table, we find that increasing the number of condition slots leads to a decrease in the model's performance on most tasks, except for the "explore run" task. We speculate that having more condition slots may result in a higher number of dimensions in the goal space, which makes the goal space less abstract and poorly generalizable. Therefore, we suggest that when applying GROOT to other environments, the hyperparameters should be carefully chosen based on the characteristics of the environment or using other parameter selection methods.

(Weakness 2) Emphasize the importance of inverse dynamic model.

Thanks for raising this question. The Inverse Dynamic Model is indeed critical in our paper because it provides necessary behavior labels for video data to compute the behavior cloning loss in subsequent steps. To avoid misunderstandings, we added a section to introduce IDM in the Appendix C of the paper. In addition, we have also emphasized this in the appropriate section of the paper, such as conclusion.

Fortunately, OpenAI made an important and surprising discovery in their VPT paper that we may only need to collect a small amount of supervised data with behavior labels to train a highly accurate inverse dynamic model, even in complex environments like Minecraft. In parallel, a work presented at ECCV2022 [1] trained an accurate IDM model in the field of autonomous driving, further validating the feasibility of training IDM models in complex domains. This allows us to migrate this algorithm to other complex open environments at a lower cost.

[1] Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining.

(Weakness 3) Rearrange the position of the images.

Thank you for pointing that. In light of your recommendation, we have rearranged the position of the images to make them close to their descriptions.

(Question 1) Analysis of baselines' performance in Figure 3 (right).

Thank you for your question. As shown in the last row of Figure 3 (right), all baselines, including VPT(bc) and STEVE-1(visual), achieved low success rates on the three tasks of "enchant sword", "dig three down and fill one up", "build snow golems". This is due to the nature of these tasks. For example,"dig three down and fill one up" is a skill that only novice players would use once before passing the night. "build snow golems" is an operation that most Minecraft players are not even aware of. Therefore, the distribution of these tasks in the collected data is extremely sparse. VPT directly models the distribution of the entire dataset, making it difficult to emerge corresponding behaviors. As for STEVE-1(visual), it also performed poorly on these tasks. We speculate that these tasks are process-oriented and a 16-frame outcome video cannot fully express them. As shown in the first plot on row 2, VPT(bc) outperformed STEVE-1(visual). We speculate that this may be because VPT(bc) was fine-tuned on an early game dataset, which may contain many clips of players using furnaces to cook food. Therefore, when a furnace was placed in the agent's hand, the agent had a higher probability of exhibiting the behavior of cooking food. However, this behavior is also difficult to express with a 16-frame outcome video.

---

We would like to extend our gratitude for your valuable feedback and queries that have helped us improve the paper. We sincerely hope that our responses have addressed your questions. We are open to answering any further queries during the discussion period, so please feel free to let us know if you have any remaining concerns or need further clarification. If we have successfully addressed your concerns, we kindly request you to consider increasing your score.

Thank the authors for the clarifications.

After reviewing the author's response and other reviews, it help me better understand the work. I understand that a lot of the details and experiments are not feasible to squeeze into the main paper.

I am thus improving my score from 6 to 8.

Dear Reviewer 9vuS,

We were wondering if you have had a chance to read our reply to your feedback. As the time window for the rebuttal is closing soon, please let us know if there are any additional questions we can answer to help raise the score!

Best, Authors

Dear Reviewer, we thank you for your comments and suggestions. Thank you for acknowledging that the research problem we investigated is very interesting, and for recognizing the clarity and other strengths of our writing in the article. Below, we will address your comments.

Clarification on problem setting.

We do not have predefined tasks for training. We would like to clarify our problem setting. The problem to be addressed in this paper is how to pre-train an instruction-following policy as well as a goal space using only large-scale video-action data or video data. (Note that if the action data is unavailable, we need to use a pre-trained inverse dynamic model to label the video with actions.) It is worth emphasizing that these videos are recordings of human players freely playing in Minecraft, and we do not have the task label. Thus, there is no concept of a "pre-defined task," and there is no information such as reward functions available. Our aim is to learn such a goal space that expresses infinite tasks in Minecraft.

(Weakness 1, Part 1) Generalization ability.

Can the authors validate the controller's generalization ability for diverse tasks by providing relevant video instructions for unseen tasks and skills? This would be of great significance for the integration of LLM-based planning agent capabilities.

Thank you for your question. Our training method is somewhat similar to GPT. OpenAI did not need to define explicit tasks or categorize their collected data by task when pre-training GPT. Instead, they crawled a large amount of noisy text data from the internet and trained the model's ability to predict the next word autoregressively. After training, GPT has the ability to solve "unseen" tasks in zero-shot. Similarly, in the training process of GROOT, we do not need to define explicit tasks or cut each video into clips according to their semantics. Instead, we train the model's ability to self-imitate on a video segment. After training, by providing GROOT with a gameplay video that describes how to complete a task, it can imitate the process described in the video to complete the specified task in Minecraft.

We can understand the tasks defined in Minecraft SkillForge as those that occur after GROOT has completed training, meaning that GROOT actually knows how to perform far more tasks than those included in SkillForge. As videos can be obtained from the internet in an unsupervised manner, we may not be able to find a behavior that GROOT has never seen, just as we would have difficulty finding a task that GPT has never encountered.

Here, we highlight two tasks that are rarely seen in the dataset: "build snow golems" and "dig down three and fill one up". The former is only known to a few players, while the latter is only used occasionally by very beginner players. GROOT achieved a success rate of 60% in the "build snow golems" task and 30% in the "dig three down and fill one up" task, which demonstrates the extent GROOT is able to perform on long-tailed or even "OOD" tasks.

(Weakness 1, Part 2) Ideas of constructing video instructions when combined with a high-level planner.

Thank you for raising this question. In recent years, there have been a number of works [1, 2] aimed at high-level planning in Minecraft. These works unanimously suggest that a low-level controller should construct a skill library for use by higher-level planners. For example, DEPS [1] uses a pre-trained goal-conditioned policy as the underlying controller, as provided in [5]. However, training such a goal-conditioned policy still requires a large amount of high-quality, task-specific demonstrations. For GROOT, we can use relevant keywords to search on YouTube or ask Minecraft players to provide some tutorials for the skills that may be called by higher-level models. In practice, we can collect about 5 gameplay videos for each skill then select the best performing one that GROOT can imitate, save it in the database. It is easy to retrieve the gameplay video from the database when the planner needs to call some skills. This has greatly reduced the cost of building a skill library. We have updated the relevant discussion in Appendix J for interested readers to check.

[1] Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents [2] Ghost in the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory [3] Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction

Dear Reviewer f4yG,

We were wondering if you have had a chance to read our reply to your feedback. As the time window for the rebuttal is closing soon, please let us know if there are any additional questions we can answer to help raise the score!

Best, Authors

Dear Reviewer,

We are very glad that we were able to solve your major problems and greatly appreciate that your score has improved. We hope that we can further clarify any doubts you may have.

---

Although "may not be able to find a behavior that GROOT has never seen," the data can be artificially divided or produced. For example, given some unreasonable behavior composed of video examples, can GROOT maintain similar behavior? This is crucial for the applicability of the method.

We defined a task of "burning oneself to death using lava," which is a very unreasonable behavior. We found that GROOT performed well on this "unreasonable" task, with 4 out of 5 episodes being successful. This further demonstrates that our method can generalize well to tasks outside the training distribution. We have put the videos here: https://anonymous.4open.science/r/lava_ano-0C5B

---

Using videos as instructions and IDM is the contribution of VPT, so I think the difference between GROOT and GATO is the addition of the KL divergence constraint in the representation space.

The video instructions mentioned in this paper refer to a way of specifying a task, which describes how a task should be completed in the form of a video.

• VPT is a model obtained by behavior cloning a large-scale trajectory dataset, which can be represented as $a_t \sim \pi(a_t | s_{1:t})$. Therefore, it does not have the ability to follow instructions and naturally cannot follow video instructions. We have tried to append a video instruction (chop tree) prefix as a prompt to the state sequence input to VPT. However, this did not prompt VPT to obtain the corresponding behavior, and the actual result was that VPT almost completely ignored this prompt.

• For GATO, as mentioned in section 5.2 of its paper, it requires collecting a large amount of expert demonstration data for new tasks and finetuning on these data to generalize to new tasks. The following is the content from section 5.2 of the GATO paper:

Ideally, the agent could potentially learn to adapt to a new task via conditioning on a prompt including demonstrations of desired behaviour. However, due to accelerator memory constraints and the extremely long sequence lengths of tokenized demonstrations, the maximum context length possible does not allow the agent to attend over an informative-enough context. Therefore, to adapt the agent to new tasks or behaviours, we choose to fine-tune the agent's parameters on a limited number of demonstrations of a single task, and then evaluate the fine-tuned model's performance in the environment.

In the GATO paper, achieving the same performance as the BC algorithm on a new task requires collecting 5000 demonstrations for that task. In contrast, for GROOT implemented with our encoder-decoder architecture, we only need to collect a minimum of 1 video to generalize to a new task, and we do not require any additional finetuning process. This means that we do not have to worry about "catastrophic forgetting" caused by finetuning on a new task.

---

Although the authors have proposed some ideas for combining with high-level planners, many special treatments are still needed in the application, such as using search engines or filtering videos, which limits the applicability.

For humans, searching for strategy guides or instructional videos recorded by other gamers on the internet is a common practice when playing games. Therefore, we believe that using a search engine or filtering video data to build a skill database is a natural method. A powerful generalist agent should have the ability to retrieve knowledge and expand its skills on the internet (although this is beyond the scope of this research). In addition, we also provide other ideas for integrating GROOT with a high-level planner. For example, by collecting text-video pairs, we can align the text instruction modality with the instruction space of GROOT. We conducted a preliminary experiment in Appendix I as an example for your reference. However, we would like to clarify that the main problem that this paper addresses is how to train an agent to understand video as a form of task description. There are many tasks that are difficult to describe in language, making video a natural choice for description.

---

We hope that the above explanations can resolve doubts you may have.

We would like to express our sincere appreciation to the reviewer for their positive feedback on the clarity and simplicity of our proposed method, as well as its potential contribution to the development of generalist agents. Thank you for your valuable insights and encouragement!

(Weakness 1) Additional experiment on text conditioning.

Thank you for your question. We believe the video instruction space is more general and some behaviors are more naturally expressed with video. It is the focus of our paper.

Meanwhile, we also agree on the importance of conditioning policy on text instructions. Although video instruction has strong expressiveness, it still requires preparing at least one gameplay video for a new task. For some tasks, such as collecting wood or stones, using natural language to specify a goal is a more natural approach. We have conducted the text-conditioning experiment, please see Appendix I.

Considering that it is not feasible for us to collect text-video data with as much diversity as STEVE-1 within a short period of time, and we did not directly utilize the initial alignment space provided by MineCLIP during GROOT training. Therefore, we chose a new alignment method, which is to directly replace the video encoder with a BERT architecture text encoder (randomly initialized) and freeze the decoder. We optimize the text encoder using behavior cloning.

We utilize the meta information in the contractor data to generate text-demonstration pairs. For example, in the task of "collect wood", we identify the moment $t$ when event "mine_block:oak_log" is triggered in the video, and we capture the frames within the range of $[t-127, t]$ to form a video clip, with "mine block oak log" assigned as its text, thus constructing a sample. Having been fine-tuned on these data, our model demonstrated some steerabilities in the text instruction space, as shown in the table below. (The term "baseline" refers to the model before being fine-tuned, while "fine-tuned" refers to the final model after fine-tuning. )

We find that the agent fine-tuned on the text-demonstration dataset shows a basic understanding of text instructions. Our method exhibits progress in tasks such as "mine grass", "mine wood", "mine stone", "mine seagrass", "pickup beef" and "mine dirt". However, it falls short in successfully completing tasks such as "mine seagrass". We speculate that this may be related to the distribution of the data, as there is much less data available for "mine seagrass" compared to the other tasks (only about 300 trajectories).

We emphasize that this experiment is very preliminary. In this experiment, the steerability of the agent fine-tuned on text instructions is still weak and it is hard to solve practical tasks. Given the limited diversity of text instructions in the provided contractor data, we don’t anticipate the model to possess any significant level of generalization with regard to language instructions. To further verify this point, one needs to collect more diverse and higher-quality text-demonstration pairs data. Anyway, this experimental result still indicates the possibility of optimizing the upstream instruction generator by leveraging the pre-trained decoder. This creates possibilities for developing more interesting applications on GROOT. Additional discussions on text-conditioning are beyond the scope of this paper, and we will leave them for future work.

(Weakness 2) Confusing notation.

Thank you for raising this question. We apologize for the confusion caused. It was a typo on our part. Our intention was to clarify that the two video encoders do not share the same set of parameters. In reality, we should use $p_\theta(g | s_{0:t})$ instead of $p_\psi(g | s_{0:t})$. We have corrected this in the paper.