Odyssey: Empowering Minecraft Agents with Open-World Skills

Q1. It would be interesting how well even smaller LMs than Llama 3 8B would perform on Minecraft under the Odyssey framework. Have any experiments of this sort been conducted?

Thanks for your constructive comments. As suggested, we have additionally compared smaller LLMs (LLaMA-3-1B and LLaMA-3-3B) on the single-round long-term planning tasks. The results in Table R1 show that larger model parameters lead to better agent performance. Specifically, LLaMA-3-3B generally performs better than LLaMA-3-1B, achieving higher success rates across the tasks. This additional experiement have been added in the revision (Page 37, Appendix F.4).

Table R1. Performance comparison of smaller LLMs on the single-round long-term planning task. All evaluation metrics are calculated only for successful tasks. $\pm$ corresponds to one standard deviation of the average evaluation over successful tasks.

---

We hope that these revisions and clarifications address your concerns. Looking forward to your re-evaluation.

---

References

[1] Automatic decomposition of planned assembly sequences into skill primitives. IEEE transactions on Robotics and Automation, 2001.

[2] Robot skills for manufacturing: From concept to industrial deployment." Robotics and Computer-Integrated Manufacturing, 2016.

[3] AutoWebGLM: A Large Language Model-based Web Navigating Agent. SIGKDD 2024.

[4] Weblinx: Real-world website navigation with multi-turn dialogue. ICML 2024.

[5] Code as Policies: Language Model Programs for Embodied Control. ICRA, 2023.

[6] ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. ICRA, 2023.

[7] NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models. AAAI, 2024.

[8] Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning. CoRL, 2023.

[9] A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges. arXiv, 2024.

[10] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.

We sincerely appreciate your positive feedback on Odyssey, particularly highlighting its valuable contribution to the community. We have carefully revised the manuscript according to your valuable suggestions. Below we address the main points raised in the review.

---

W1. The proposed framework has limited novelty. Decomposing complex decision-making tasks with hand-engineered skill libraries has a very long history in robotics [1,2].

Thanks for your valuable comments. (1) We would like to emphasize that our contribution does not lie in technical novelty. This is why we select "datasets and benchmarks" as the primary area in the OpenReview submission, which is more suited for contributions like ours that offer new datasets, benchmarks, and tools to the research community. We believe that one of our significant contributions is indeed the engineering domain-specific skill library, which represents a meaningful addition to the Minecraft agent community. Developing this library requires significant time and effort in design and debugging, making it a valuable resource for researchers. By leveraging our skill library, even open-source LLMs can enable open-world exploration within the Minecraft environment.

LLM-based Voyager advances exploration using skill generation but relies on GPT-4, incurring substantial costs (reaching thousands of dollars per experiment). This expense can be prohibitive for many researchers. On the contrary, our agent can call upon the proposed skill library, achieving performance comparable to Voyager with GPT-4, but using only 8B LLMs.

(2) The skill library is not the only contribution of our work. The proposed Odyssey consists of three key contributions: (a) a fine-tuned LLaMA-3 model trained on a large-scale question-answering dataset; (b) an interactive agent equipped with an open-world skill library; (c) a new agent capability benchmark encompassing a variety of tasks. We have open-sourced all parts of ODYSSEY (see supplementary material) and will continuously update the repository. We hope this will enable other researchers to build upon our work, fostering further innovation and progress in the development of autonomous agents.

W2. The Odyssey framework is designed specifically for Minecraft. Agentic performance is significantly boosted through the careful design of useful, hand-engineered low-level skills. As a result, it is unclear to what extent good LM performance on Minecraft with Odyssey would transfer to other, more practical open-world environments like Web navigation.

Thanks for your insightful comments. We have provided a discussion on migrating Odyssey to other domains in the revision (Page 16, Appendix B), as follows:

Discussion on Migrating Odyssey to Other Domains

The skill library designed for Minecraft is built with modularity and generalizability in mind, allowing for potential adaptation to other domains such as web navigation [3, 4], robot manipulation [1, 2, 5, 6], robot navigation [7, 8], and other game-playing environments [9]. These skills abstract underlying actions and focus on high-level interactions, allowing them to be adapted to different environments by redefining low-level actions without changing the overall structure of the skill library. Even without direct API access, basic action spaces (e.g., keyboard and mouse operations in games, or movement operations in robotics) can be employed to construct primitive skills. Prior research in robotic manipulation, including CaP [5] and ProgPrompt [6], demonstrates how primitive skills such as picking and placing objects or opening containers can be built from basic actions. Moreover, we believe that the concept of "skills" should extend beyond code APIs to include knowledge from various sources. For example, handbooks can provide informational segments treated as skills, retrievable by LLMs using techniques like retrieval-augmented generation [10], enhancing decision-making.

To fine-tune the LLaMA-3 model for the Minecraft agent, we crawled the Minecraft Wiki and used a GPT-assisted approach to generate an instruction dataset. Researchers in other domains can replicate this process to create their own instruction datasets. To facilitate this, we have open-sourced our Minecraft Wiki crawler on Github, which can be easily modified to crawl similar Wiki websites for other domains. Additionally, our benchmark tasks evaluate agent performance from three perspectives: long-term planning, dynamic-immediate planning, and autonomous exploration. These dimensions effectively assess the capabilities of open-world autonomous agents. Researchers in other domains can adopt these perspectives to design comprehensive evaluation tasks for their needs.

We greatly appreciate the reviewer for the insightful comments on Odyssey, which helped improve the quality of the paper significantly. We have carefully revised the manuscript according to your valuable suggestions. Below we address the main points raised in the review.

W1. ODYSSEY’s pipeline is highly similar to existing frameworks such as Voyager, Optimus-1[1], and ADAM[2].

Thanks for your comment. It is worth noting that our focus is not to design a new LLM-based agent architecture. Instead, this work aims to provide a comprehensive framework for developing and evaluating autonomous agents in open-world environments, enabling them to explore the vast and diverse Minecraft world.

(1) Existing efforts mainly focus on solving basic programmatic tasks, like material collection and tool-crafting. This limitation stems from the narrowly defined set of actions available to agents, requiring them to learn effective long-horizon strategies from scratch. Thus, discovering diverse gameplay opportunities in the open world becomes challenging. LLM-based Voyager advanced exploration using skill generation but relied on GPT-4, incurring substantial costs (reaching thousands of dollars per experiment). This expense can be prohibitive for many researchers. In response, we develop a skill library that agents can call upon, achieving performance comparable to Voyager with GPT-4, but using only 8B LLMs. This makes our approach significantly more accessible and cost-effective. Our work facilitates deeper research within the Minecraft environment, extending beyond basic programmatic tasks. We believe this potential for broader exploration represents a major contribution to the research community.

(2) Please note that Optimus-1 and ADAM are our concurrent work (Optimus-1 was published on arXiv on Aug 7, 2024, and updated on Oct 21, 2024; ADAM was published on arXiv on Oct 29, 2024, while the ICLR submission date was Oct 1, 2024).

W2. ODYSSEY relies on predefined primitive skills, which were generated by GPT-4, whereas GPT-4 itself can directly write JavaScript programs based on Mineflayer. This approach of relying on primitive skills limits the agent’s ability to perform more complex and open-ended tasks, such as building.

Sorry for the misunderstanding.

(1) Our skill library not only includes primitive skills but also features 183 compositional skills that Voyager lacks. Compositional skills encapsulate primitive skills into higher-level tasks, addressing various basic programmatic tasks like mineDiamond and craftIronPickaxe. While Voyager builds its skill library by allowing the LLM to create skills from a limited set of primitive skills, this leads to unstable skill generation and high learning costs. In contrast, Odyssey offers an open-world skill library with both primitive and compositional skills, enabling LLM-based agents to efficiently generate complex policies for broader exploration and more complex tasks.

(2) We believe that providing a set of skills is not limited but rather essential for practical open-world scenarios. In real-world applications, it is unrealistic to expect an agent to learn all tasks from scratch. Pre-existing skills allow the agent to efficiently handle simpler tasks, thereby enabling it to focus on more complex challenges. Please note that our skill library also includes basic building-related skills, such as crafting and placing, which the agent can use to complete complex building tasks. It is important to emphasize that the tasks in our benchmark cannot be simply solved by any single skill. Instead, they require a sophisticated combination of multiple skills, ensuring that the benchmark remains challenging.

W3. On programmatic tasks, ODYSSEY does not demonstrate a broader task range compared to baselines, remaining at the diamond level, already achievable by Voyager. What about more difficult tasks?

Thanks for your comment. Early Minecraft research often focuses on basic programmatic tasks, treating the ObtainDiamond task as the main challenge. This approach only assesses the ability of agents to prioritize crafting steps within a limited task space, rather than their potential for developing complex and diverse solutions. Therefore, this paper does not focus on basic programmatic tasks; we use them merely to simply verify the effectiveness of the proposed skill library. Our benchmark introduces three distinct tasks to evaluate the planning capabilities of agents in Minecraft: long-term planning tasks, which assess the agent's ability to formulate and execute comprehensive plans over extended periods; dynamic-immediate planning tasks, which require agents to adapt their plans dynamically based on immediate environmental feedback; and autonomous exploration tasks, which evaluate the agent's ability to explore the Minecraft world independently, without specific goals.

W4. The comparisons shown in Table 3 are unfair, as DEPS and VPT use keyboard and mouse as action spaces, rather than JavaScript code, and VPT additionally utilizes visual observation. This is fundamentally different from ODYSSEY, which uses privileged information as its observation space, making such comparisons invalid.

Thanks for your valuable feedback. We have removed Table 3 in the revision. In our experiments on the open-world skill library (Page 6, Section 5.1), we have updated the experiments to focus solely on testing the effectiveness of directly invoking different skills (without using an agent, and therefore not directly comparing with existing agent methods). The results in Table 2 demonstrate that our open-world skill library efficiently handles basic programmatic task.

W5. The authors fine-tuned LLaMA-3 on a supplementary dataset (Minecraft Wiki) to create MineMA, but in Tables 4 and 5, the comparison is made against open-source models of equivalent size that lack Minecraft-specific knowledge, resulting in weaker performance. I suggest comparing MineMA with models like GPT and Claude, which possess robust Minecraft knowledge, to demonstrate the significance and efficacy of the additional fine-tuning.

Thanks for your valuable suggestion. We have additionally compared our MineMA with powerful GPT-4o. The results in Table 4 show that our MineMA can achieve performance similar to that of GPT-4o. The corresponding tables and analysis have been updated in the revision (Page 8--9, Section 5.2.2).

Table 4. Performance comparison of different models on the dynamic-immediate planning task.

W6. Several related works were not cited, including: [1] Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks [2] ADAM: An Embodied Causal Agent in Open-World Environments [3] OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents, NeurIPS 2024 [4] Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds, ICLR 2024

Sorry for the missing related works. We have cited these works in the revision (Page 9, Section 6 Related Works).

---

We hope that these revisions and clarifications address your concerns. Looking forward to your re-evaluation.

We greatly appreciate the reviewer for the insightful comments on Odyssey, which helped improve the quality of the paper significantly. We have carefully revised the manuscript according to your valuable suggestions. Below we address the main points raised in the review.

W1. In the rebuttal, I do expect a comprehensive review of how the contribution presented in the manuscript can be more significant than these for building better open-world agents.

Thanks for your valuable comments. We would like to emphasize that our contribution does not lie in technical novelty. This is why we select "datasets and benchmarks" as the primary area in the OpenReview submission, which is more suited for contributions like ours that offer new datasets, benchmarks, and tools to the research community. The proposed Odyssey consists of three key contributions:

• An interactive agent equipped with an open-world skill library. Existing efforts mainly focus on solving basic programmatic tasks, like material collection and tool-crafting. This limitation stems from the narrowly defined set of actions available to agents, requiring them to learn effective long-horizon strategies from scratch. Thus, discovering diverse gameplay opportunities in the open world becomes challenging. LLM-based Voyager advanced exploration using skill generation but relied on GPT-4, incurring substantial costs (reaching thousands of dollars per experiment). This expense can be prohibitive for many researchers. In response, we develop a skill library that agents can call upon, achieving performance comparable to Voyager with GPT-4, but using only 8B LLMs. This makes our approach significantly more accessible and cost-effective. Our work facilitates deeper research within the Minecraft environment, extending beyond basic programmatic tasks. We believe this potential for broader exploration represents a major contribution to the research community.

• A fine-tuned LLaMA-3 model trained on a large-scale question-answering dataset. Our dataset involves 390k+ instruction entries derived from the Minecraft Wiki. In contrast, the Wiki dataset released by MineDojo only collects Minecraft Wiki pages, without refining the content and generating Q&A pairs for model training. STEVE introduces a non-public dataset with 20k+ Q&A pairs, which is smaller than our dataset in terms of scale and diversity. The OmniJARVIS dataset is currently not public, and without knowing its exact size, we cannot make a comparison. It is notable that OmniJARVIS is our concurrent work (it was published on arXiv on Jun 27, 2024, and updated on Oct 31, 2024, while the ICLR submission date was Oct 1, 2024).

• A new agent capability benchmark encompassing a variety of tasks. Existing benchmark with programmatic tasks only assess the ability of agents to prioritize crafting steps within a limited task space, rather than their potential for developing complex and diverse solutions. Our benchmark introduces three distinct tasks to evaluate the planning capabilities of agents in Minecraft: long-term planning tasks, which assess the agent ability to formulate and execute comprehensive plans over extended periods; dynamic-immediate planning tasks, which require agents to adapt their plans dynamically based on immediate environmental feedback; and autonomous exploration tasks, which evaluate the agent ability to explore the Minecraft world independently, without specific goals. It is notable that GROOT-2 is also our concurrent work, which is under anonymous review in ICLR 2025.

W2. The results in table 3 should be more carefully examined, as two of the three baselines indeed employ end-to-end control rather than scripted skills. Without an ablation on this, it cannot justify the effectiveness of the proposed method, at least on programmatic tasks.

Thanks for your valuable feedback. We have removed Table 3 in the revision. In our experiments on the open-world skill library (Page 6, Section 5.1), we have updated the experiments to focus solely on testing the effectiveness of directly invoking different skills (without using an agent, and therefore not directly comparing with existing agent methods). The results in Table 2 demonstrate that our open-world skill library efficiently handles basic programmatic task. Simple tasks achieve near-perfect success within five minutes. Even for difficult tasks like obtaining a diamond, success rates rise from 21.7% at five minutes to 92.5% at ten minutes, highlighting the effectiveness of the skill library.

---

We hope that these revisions and clarifications address your concerns. Looking forward to your re-evaluation.

We greatly appreciate the reviewer for the insightful comments on Odyssey, which helped improve the quality of the paper significantly. We have carefully revised the manuscript according to your valuable suggestions. Below we address the main points raised in the review.

---

W1. The paper repeatedly emphasizes that “our focus is not to design a new LLM-based agent architecture.” However, a significant portion is still dedicated to detailing the agent architecture, even listing it as part of the contribution. Since this architecture is not novel, it would be better suited to the appendix.

Sorry for the confusion. As stated in the Introduction (Page 2, Line 105--106), this work aims to provide a comprehensive framework for developing and evaluating autonomous agents in open-world environments. We believe that a brief explanation of the agent architecture is crucial for understanding how our framework integrates with the existing architecture to achieve the results we present. To address your concern, we have condensed the section on the agent architecture to approximately half a page in the revision (Page 4, Section 2.2).

W2. Given that the focus is not on a "new LLM-based agent architecture," performing an ablation study on a standard architecture seems less meaningful.

Sorry for the confusion. As mentioned in our response to W1, one of our main contributions is the evaluation of autonomous agents. Our LLM-based agent employs a planner-actor-critic architecture to facilitate task decomposition, skill execution, and performance feedback. This is a widely used general framework, and since our goal is to assess the agent performance on the proposed benchmark, it is essential to test the effectiveness and robustness of the different components within the agent. This provides a necessary reference point for understanding how each part contributes to the overall performance.

W3. The comparison in Table 3 is inherently unfair. The VPT model operates in the native, unmodified environment with RGB output and mouse and keyboard controls, while GITM and the proposed work use Mineflayer as a controller.

Thanks for your valuable feedback. We have removed Table 3 in the revision. In our experiments on the open-world skill library (Page 6, Section 5.1), we have updated the experiments to focus solely on testing the effectiveness of directly invoking different skills (without using an agent, and therefore not directly comparing with existing agent methods). The results in Table 2 demonstrate that our open-world skill library efficiently handles basic programmatic task.

W4. Fine-tuning on Minecraft-specific knowledge is expected to improve performance compared to large, untuned models, so this result is unsurprising.

Thanks for your valuable comments. We would like to emphasize that our contribution does not lie in technical novelty. This is why we select "datasets and benchmarks" as the primary area in the OpenReview submission, which is more suited for contributions like ours that offer new datasets, benchmarks, and tools to the research community. Therefore, one of our main contributions is providing the fine-tuned LLaMA-3 model and a comprehensive question-answering dataset. The experiments are conducted merely to validate the effectiveness of the model and data we provide.

Existing researchers can utilize our fine-tuned LLaMA-3 model to construct their own Minecraft agents. Additionally, they can leverage our dataset to further fine-tune models tailored to their specific requirements. We have also open-sourced our web crawler program on GitHub, which was used to collect data from Wikis. Researchers can modify this program to crawl data relevant to their needs.

---

We hope that these revisions and clarifications address your concerns. Looking forward to your re-evaluation.

We are deeply honored to receive your response and your recognition of our work, and we sincerely appreciate your thoughtful feedback.

(1) We consider the choice of whether to use MineFlayer with text-based game as two different paradigm for building a Minecraft agent, essentially differing in how the observation space and action space is defined. We believe that providing a set of skills is not a limitation but rather essential for practical open-world scenarios. In real-world applications, it is unrealistic to expect an agent to learn all tasks from scratch. Pre-existing skills allow the agent to efficiently handle simpler tasks, thereby enabling it to focus on more complex challenges. We believe that one of our significant contributions is indeed the engineering domain-specific skill library, which represents a meaningful addition to the Minecraft agent community. Developing this library requires significant time and effort in design and debugging, making it a valuable resource for researchers. By leveraging our skill library, even open-source LLMs can enable open-world exploration within the Minecraft environment.

LLM-based Voyager advances exploration using skill generation but relies on GPT-4, incurring substantial costs (reaching thousands of dollars per experiment). This expense can be prohibitive for many researchers. On the contrary, our agent can call upon the proposed skill library, achieving performance comparable to Voyager with GPT-4, but using only 8B LLMs.

(2) We would like to emphasize that our contribution does not lie in technical novelty. This is why we select "datasets and benchmarks" as the primary area in the OpenReview submission, which is more suited for contributions like ours that offer new datasets, benchmarks, and tools to the research community. The proposed Odyssey consists of three key contributions: (a) a fine-tuned LLaMA-3 model trained on a large-scale question-answering dataset; (b) an interactive agent equipped with an open-world skill library; (c) a new agent capability benchmark encompassing a variety of tasks. We have open-sourced all parts of ODYSSEY (see supplementary material) and will continuously update the repository. While our work may not directly align with your interests, we hope this will enable other researchers to build upon our work, fostering further innovation and progress in the development of autonomous agents.

We sincerely look forward to your reevaluation of our work and would very appreciate it if you could raise your score to boost our chance of more exposure to the community. Thank you once again :)