MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory

We sincerely appreciate the reviewer for positive feedback about our paper and experiment setup and valuable comments for our paper’s weakness.

Aren’t the tasks, considering only navigation-like tasks, too simple?

We sincerely thank the reviewer for thoughtful feedback on task complexity. We would like to emphasize that Minecraft has long been regarded as a benchmark environment for general-purpose embodied AI due to its inherently complex and dynamic environment. In our experiments, the ABA-Sparse tasks in Section 4.2 and the Long-Instruction task in Section 4.4 involve interactions with dynamic entities such as cows and sheep, demonstrating that these tasks go beyond mere navigation challenges. Additionally, we would like to highlight our core contribution: the demonstrated failure of Steve-1 in instruction-following scenarios, as shown in Figure 1 of the manuscript, and further elaborated in Common Response CR1.

For further clarity, we have provided two demonstration videos: one task showcasing an ABA-Sparse task (milk-leaves-milk) and another illustrating long-horizon planning task using LLM with Mr. Steve (make a bed). These videos are available at https://imgur.com/a/brKrQhL.

What are the advantages of the proposed method over classic SLAM?

We thank the reviewer for their constructive comments regarding the relation between with our exploration method and methods from robotics domain. We answered this in Common Response CR3.

Limitations due to the lack of memory update/discard mechanism

Thanks for the constructive feedback. We acknowledge that PEM has limitations on storing dynamic entities. Although Mr.Steve shows promising performance in tasks requiring interaction with those dynamic entities (e.g., ABA-Sparse Milk-Leaves-Milk task), the memory in current PEM can become stale over time, posing difficulties in discarding or updating the memory. We discussed this issue in Common Response CR0.

Error Bars in Success Rate Plots

We appreciate the valuable feedback regarding statistical significance analysis. We acknowledge that including error bars is important for analyzing statistical significance. In our experiments, we followed the evaluation procedures of previous works [1, 2], which evaluate tasks in Minecraft using 30 to 70 seeds per task. To ensure robustness, we went beyond this standard by running all tasks with 100 different seeds to compute the success rates. However, due to the slow simulation speed of Minecraft and computational constraints, it was not feasible to perform additional runs beyond this setup during the discussion period. Nevertheless, we understand the importance of this aspect and will include results with more seeds and corresponding confidence intervals in the camera-ready version of the paper.

References

[1] Zihao Wang, et al. “Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.” NeurIPS. 2023.

[2] Zihao Wang, et al. “JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models.” NeurIPS Workshop on Agent Learning in Open-Endedness. 2023.

We thank the reviewer for highlighting the strengths of our work, including the technical soundness, clear algorithm description, and strong performance in long-horizon tasks.

Many frames in PEM do not constitute meaningful events and PEM may have unnecessary memory storage

Thank you for highlighting this important aspect of our work. We appreciate your observations about the potential limitations of PEM. In PEM, it first segments the agent’s trajectory into place-based clusters, where each cluster contains observations from nearby locations and similar directions. This ensures that the each place cluster is spatially organized and includes observations relevant to specific locations. Then, event clusters are formed from the place cluster by using MineCLIP representation. However, as the reviewer mentioned, many frames in event clusters during the tasks do not constitute meaningful events. We agree that the term “event memory” might imply high-level semantic events, which is not always the case here. Our focus was on defining an event memory suitable for low-level controllers, which can store both semantic high-level events (e.g., burning zombies) and visually novel frames (e.g., cow in the forest) that might still be useful for future tasks. While this approach may include some redundant scenes, we opted for a structure that maximizes the chances of retaining frames potentially useful for subsequent tasks. Moreover, we recognize that optimizing memory to further reduce redundancy without losing important frames is a worthwhile direction for future work. To this end, we implemented a simple version where only center embeddings of event clusters are retained, which did not degrade performance significantly. Details are provided in Appendix P.

We recognize that referring to the clustered experience frames as “event memory” may not align perfectly with the traditional meaning of “events.” The current version, based on place-observation memory, aligns more closely with the concept of What-Where-When memory as suggested in the title. In future versions, we plan to replace “event memory” with a more appropriate term, “3W-Memory.” However, to avoid causing confusion with the existing content, we have decided not to apply this change yet and will update it in the camera-ready version.

Rule-based Exploration does not leverage knowledge in PEM

Thank you for your constructive feedback. We acknowledge that the current hierarchical exploration in Mr.Steve selects positions with lowest visit frequency from visitation map, which does not utilize knowledge stored in PEM. We notice that task-conditioning is crucial for effective exploration. For instance, the agent should prioritize exploring a forest rather than a desert when searching for a tree. To address this, we have developed an advanced hierarchical exploration strategy that incorporates the same mechanisms used in exploitation. Further details are provided in Common Response CR2.

Further explanations on Long-Horizon tasks in Section 4.4

Thank you for your constructive feedback. We agree that the experimental results in Section 4.4 require further explanations, and we appreciate the opportunity to elaborate.

In the Long-Instruction task, the agent is continuously assigned random “Obtain $X$” tasks, where $X$ could be water, beef, wool, log, dirt, or seeds. Resources such as beef and wool are located in visually similar forest-like areas but at different places. Mr.Steve and Steve-PM effectively retain task-relevant information for these distinct locations, whereas Steve-EM clusters visually similar events from different places into the same event cluster, potentially losing task-relevant frames. Consequently, Mr.Steve and Steve-PM solve over 80 tasks, while Steve-EM solves only around 50 tasks.

In the Long-Navigation task, the task consists of exploration phase of $16$K steps and a task phase. In the exploration phase, the agent observes six events in different places: 1) burning zombies, 2) river, 3) sugarcane blow up, 4) spider spawn, 5) tree, and 6) house, spending $2$K steps at each place. In the task phase, the image goal is continuously given randomly selected from the frames in the early steps of the event. Note that event 1, 3, 4 are dynamic events which only occur in the early steps when the agent arrives at the event-occuring places. When the agent is in task phase, Mr.Steve and Steve-EM can maintain all events in the memory through event clustering (We note that each events are visually distinct enough for event clustering), while Steve-PM, with its FIFO-based place clusters, loses early steps of dynamic events. As a result, Mr.Steve and Steve-EM solved around 70 tasks, while Steve-PM solved less than 20 tasks.

We sincerely thank you for your thoughtful feedback and for updating the score. We have initiated a comparison study between PEM and SLAM-based approaches as suggested. While we are working diligently to complete these additional experiments, the time constraints of the extended discussion period may make it challenging to obtain results before the deadline. Nevertheless, we will make every effort to conduct this analysis and will update you with our findings as soon as they become available. We greatly appreciate your constructive comments, which have helped strengthen our work significantly.

We would like to extend our sincere thanks to Reviewer 29NL for their positive and encouraging feedback on our manuscript. Your constructive comments have been instrumental in refining our paper.

More Explanations on MineCLIP

As suggested, we added “, which is a CLIP model trained on web videos of Minecraft gameplay and associated captions.” in Line 232 in the updated manuscript. Thanks for the suggestion.

Details on DP-Means Algorithm

Thanks for the constructive feedback. We acknowledge that some details with DP-Means algorithm is insufficient. DP-Means algorithm is a Bayesian non-parametric extension of the K-means algorithm based on small variance asymptotic approximation of the Dirichlet Process Mixture Model. It doesn't require prior knowledge on the number of clusters $K$. To run this algorithm, we first set the initial number of clusters $K'$ and cluster the data with K++ Means initialization ($K'$ can be $1$), then DP-Means algorithm automatically re-adjust the number of clusters based on the data points and cluster penalty parameter $\delta$. Thus, DP-Means algorithm behaves similarly to K-means with the exception that a new cluster is formed whenever a data point is farther than $\delta$ away from every existing cluster centroid. We added the details of DP-Means in Appendix E. Event Cluster Details (Line 1145).

Using center embedding for the first top-k cluster memory read

Thank you for pointing out this aspect. We proposed one method for structuring the memory system, but it is possible to enhance this by setting multiple center embeddings per cluster (e.g., using an additional K-Means) to handle queries. While this would increase computational cost linearly with the number of center embeddings, the efficiency of the query operation can still be maintained as long as the number of center embeddings remains significantly smaller than the total data points within a cluster. In our experiments, we found that using a single center embedding per cluster was sufficient.

Regarding your specific concern, we appreciate your insightful example. In our approach, we assume that the agent’s position and orientation are known, which helps ensure that each cluster in the place memory contains observations from nearby locations and similar directions (More details in Appendix E, Line 1158). However, as the reviewer pointed out, a single center embedding might not fully represent the agent’s focus within a specific space. This is because a single space can host various events, and in place memory, the center embedding is selected as the observation closest to the geometric center of a cluster. To address this limitation, our proposed Place Event Memory (PEM) incorporates event-based clustering within a single spatial area. By doing so, the center embedding for each cluster corresponds to a distinct and semantically meaningful event, improving the relevance of the retrieved information.

Unclear Novelty and Significance of the paper

Thank you for your thoughtful review and for recognizing the strong execution of our method. Below, we address the concerns about novelty and significance.

Count-based Exploration and Goal-conditioned Navigation We acknowledge that count-based exploration is a well-established method, and we adopted it based on prior works in SLAM. For goal-conditioned navigation, we want to emphasize differences of VPT-Nav from previous works. In Steve-1, and GROOT [1], VPT is finetuned for goal-conditioned behavior cloning (supervised learning). In DECKARD [2], and PTGM [3], VPT is finetuned with adapter [4] for single task with reward (RL). We found that naively combining goal-conditioning from Steve-1, and RL finetuning from DECKARD showed suboptimal navigation behavior. Thus, we came up with different conditioning and recently proposed LoRA adaptor for RL finetuning. This resulted in VPT-Nav’s optimal navigating behaviors outperforming goal-conditioned navigation policy from Plan4MC (RL from scratch). We believe this approach provides insights into improving goal-conditioned navigation for foundation policy models. Further details are provided in Appendix L.

Memory System In systems like Steve-1 that uses low-level controllers based on Transformer-XL architectures, recalling observations from more than a few thousand steps earlier is highly inefficient. To address this limitation, we proposed a Place Event Memory (PEM) that efficiently stores novel events from visited places, enabling sparse and effective sequential task-solving.

We express our gratitude to the reviewer for the recognition of novelty of the method and the strong results of our experiments.

Can Event Clusters in PEM Capture Dynamic Events?

Yes. In PEM, each of newly created clusters from DP-Means are either merged to existing event clusters or allocated as new event cluster for each place cluster. Whenever additional $100$ frames are stored in the place cluster, DP-Means [1] is applied and clusters are created. As the reviewer mentioned, each created cluster is merged to one of event cluster if the similarity of MineCLIP [2] representation of their center embeddings are higher than $c$, or allocated as a new event cluster otherwise. We used $c=73.5$ (in Appendix E.4) in all experiments for consistency.

In burning zombies example, let’s assume zombies spawn at night in some place (confined in fence as in Figure 6(b)), and zombies burn and disappear in the morning. During night, PEM generates single event cluster since scenes during night are similar and resulting clusters from DP-Means are merged. However, when zombies burn in the morning, PEM generates new event cluster for burning zombies’ scenes, and another event cluster for scenes that zombies disappear. We found that MineCLIP representation works reasonably well to cluster semantically different events with proper $c$. We also note that PEM includes the game time in the memory, as the experience frame $x_t= \\{ e_t,l_t,t \\}$ is stored in the memory as stated in section 3.1. Regarding this issue, we gave a more concrete answer in Common Response CR0.

Balance between Strong and Weak Separator for Efficient Exploration

We thank the reviewer for their insightful comments regarding the need for advancements on hierarchical episodic exploration. Following you feedback, we implemented the advanced version of high-level goal selector for more efficient exploration. See Common Response CR2 for more details.

Additional Experiment for ABA-Sparse Tasks with Memory Constraints

We thank the reviewer for supportive feedback on ABA-sparse task in section 4.2. As suggested, we evaluated Mr.Steve, Steve-EM, Steve-PM, Steve-FM in 6 ABA-sparse tasks when there is a limitation in memory capacity. We tested different memory capacities (0.1K, 0.5K, 1K, 2K) for each model. In Figure 17 in Appendix O, we found that Mr.Steve maintained its performance even with low memory capacities. In case of Steve-FM, we observe that its performance decreases as the memory capacity gets small since it loses the experience frames for solving first task A. Interestingly, we found that Steve-PM, and Steve-EM showed degraded performance on Beef-Log-Beef and Beef-Leaves-Beef tasks when memory capacity is 0.1K. This indicates the robustness of PEM to memory capacities in ABA-Sparse tasks. For further details, we updated the paper with the results in Appendix O.

In the example where meat needs to be found, what happens if navigating back to the location where cows were previously seen does not have cows anymore? Is there a way to update or forget information in the PEM?

We answered this in Common Response CR0.

In Figure 5, are the results, particularly comparing Mr.Steve, Steve-EM, Steve-PM, and Steve-FM, statistically significant (particularly for Wool-Dirt-Wool and Milk-Sand-Milk)?

Thank you for the constructive feedback. We acknowledge the reviewer’s concerns regarding the statistical significance of the results in Figure 5. The purpose of the ABA-Sparse tasks in Figure 5 was to demonstrate the significant performance improvements of Steve-1 when augmented with memory. Upon reflection, we recognize that including ablations of Mr. Steve with different memory variants may not be good choice. Instead, we highlighted the benefits of PEM over various memory variants in Section 4.3 and Appendix O, and added this content in Figure 5 caption.

Additionally, we understand that the notation for Mr. Steve’s memory variants may have caused some confusion. Since these memory variants are all introduced in our paper, we will revise their names in the camera-ready version as follows: Steve-PM → Mr. Steve-PM, Steve-EM → Mr. Steve-EM, and Steve-FM → Mr. Steve-FM.

References

[1] Or Dinari, et al. “Revisiting DP-means: Fast scalable algorithms via parallelism and delayed cluster creation.” UAI. 2022.

[2] Linxi Fan, et al. “Minedojo: Building open-ended embodied agents with internet-scale knowledge.” NeurIPS. 2022.

Thank you for taking the time to comment on my initial review. Most answers are compelling to me; however, the intuition behind the event clusters for dynamic events is still a little unclear to me. In your example detailing the burning zombies, I understand that PEM would generate new clusters as them burning and eventually disappearing would be a significant event. However, as far as I understand right now, this would require the agent to actually see the events to properly model them. In your task in which you need to retrieve items (e.g., Milk-Sand-Milk), if such a task includes a dynamic event (assume you want to find burning zombies), would MrSteve be able to anticipate certain locations? Assume two days ago, you saw that zombies burned in the morning at a particular location, but that location is far away now. Later on, you found, during last night, a location with zombies, but it was night, so they weren't burning (yet). Now, your task is to find burning zombies. Where would MrSteve go? The far-away zombies in the hopes of making it there in time (as you said, time is important for your clustering), or would it go to the significantly closer zombies that it hasn't seen burning yet but can transfer that knowledge from the faraway ones that the close ones will burn soon?

Thank you for clarifying your question further. We now have a much clearer understanding of what you meant by 'dynamic event'. We think this is not a memory problem but rather an inference or prediction problem, where the agent needs to learn a world model that reflects the knowledge that night zombies turn into burning zombies in the morning. The current version of our PEM simply stores what it has observed, but does not fill the gaps between observations through reasoning. Thus, Mr.Steve would navigate to the far-away zombies which the burning is observed and not reaching to nearby zombies in your example.

Nevertheless, we believe this question is very thought-provoking and interesting to consider for future work. One way to handle such a situation would be to have the LLM-based high-level planner read events from the PEM to reason about the fact that nearby night zombies will turn into burning zombies. We think this represents an interesting spatiotemporal reasoning challenge where GPT-like chain-of-thought reasoning could be helpful in the spatiotemporal space.

Additionally, we could introduce temporal imagination capabilities similar to those used in planning models like Dreamer. Specifically, we could pick an event (e.g., night zombie) from PEM and use a learned world model to rollout its future state. If the model has learned effectively, it would predict the transformation into a burning zombie, which can let Mr.Steve navigate to nearby zombies in the example. In both approaches, however, this task seems more appropriate for the higher-level planner or reasoner rather than the low-level controller like Mr.Steve.

Thank you for the additional answer and thoughts on how your work could be used to make even better agents that can reason over the memory you are proposing in the current work. I fully recognize that this is out of the scope of this work; however, I think this would make for an interesting paragraph in future work. Either way, I will update my score to accept.

From the above table, we found that LLM-based agent shows higher success rates when using Mr.Steve as low-level controller in all tasks. This is because most of the tasks require doing the same sub-task multiple times (e.g., mining a log three times), and memory in Mr.Steve can efficiently memorize the previous task and re-do it again. More elaborations and results can be found in Appendix M. Also, we provide the video of LLM-based Agent with Mr.Steve that solves ‘make bed’ task in https://imgur.com/a/brKrQhL.

CR2. Demonstrating Task-Conditioned Hierarchical Episodic Exploration

We sincerely appreciate the valuable insights provided by 29NL, UxTw and 86n7 regarding episodic exploration methods. Building upon these insights, we implemented and evaluated a task-conditioned hierarchical episodic exploration method that leverages knowledge stored in PEM.

In task-free hierarchical episodic exploration (method proposed in the original manuscript), it selects the next goal randomly from least-visited locations. This does not utilize knowledge stored in PEM. For instance, the agent should prioritize exploring a forest rather than a desert when searching for a tree. To tackle this, task-conditioned hierarchical episodic exploration selects the next goal with highest task-relevance from least-visited locations. Specifically, task-relevance map is constructed to represent the task-relevant scores on agent’s trajectory. We used MineCLIP to calculate the task-relevant scores for each event cluster by computing the alignment score between center embedding of each cluster and text prompt.

To see whether task-conditioned exploration benefits by using knowledge in PEM, we evaluated Mr.Steve with both exploration methods (task-conditioned and task-free) on three ABA-Sparse tasks. In the below table, success rate and agent’s exploration time are given for two exploration methods. While the performance improvement from task-conditioned exploration was not substantial, it significantly reduced exploration time by approximately 300~400 steps compared to the task-free method, indicating a more efficient exploration. For further clarity, we have updated the implementation details and full experimental results in Appendix N.

CR3. What are the advantages of the proposed exploration method compared to those in the robotics domain?

We are grateful for the insights provided by reviewers UxTw, and 86n7 regarding the relation between our proposed exploration method and those in robotics domain. We agree that the hierarchical exploration method we propose could be seen as simplified version of SLAM techniques commonly used in robotics. However, when comparing methods from robotics with ours, it is important to consider the distinction between high-level goal selector and low-level goal-conditioned navigation policy.

At the high level, SLAM-based approaches typically use a top-down map [3] or construct a topological map [4, 5, 6] of the agent’s trajectory to propose the next exploration location or the optimal path between two locations. For instance, Robohop employs foundation models like SAM and DINO to create topological maps, and this method could complement our high-level Count-based exploration strategy.

At the low level, however, the goal-conditioned navigation policies in these approaches often rely on RL policies trained from scratch [3] or built-in point navigation controllers for real robots [3, 5]. RoboHop, in particular, uses pixel-level heuristic navigation, which may not be suitable for complex and interactive environments like Minecraft. In scenarios where agents face challenges such as being pushed by enemies, crossing rivers or mountains, or navigating around sheep and cows, massive prior knowledge of the environment is essential for successful navigation. To address this, we proposed VPT-Nav, a goal-conditioned navigation policy built on VPT, a foundation policy model trained with human demonstration data. By applying adjustments such as LoRA adaptors and optimizing the placement of goal conditioning, we achieved significant improvements over previous VPT fine-tuning methods. VPT-Nav also outperformed RL navigation policies trained from scratch and heuristic policies used in Plan4MC, as detailed in Appendix L. We believe our approach to combining foundation policy models with goal-conditioned fine-tuning can be effectively leveraged with high-level goal selector in robotics.