ADAM: An Embodied Causal Agent in Open-World Environments

What are possible limitations of the method? (the paper does not mention any)

6. Open-source MLLMs are currently insufficiently accurate to provide detailed information for complex tasks, as discussed in my response 7 to Reviewer GNQs. Existing MLLMs primarily assist in observing peripheral environments, which has been validated through our ablation experiments. However, human feedback remains necessary for Minecraft tasks requiring high accuracy, as also noted by VOYAGER [1]. To our knowledge, relying solely on vision (without metadata) for complex tasks like building remains unfeasible, and we are committed to advancing in this direction, including modeling causal relationships in such tasks. ADAM's decision-making process relies on its memory, the causal graph it constructs, and the reasoning ability of LLMs, with MLLM playing only a supporting role.

What do the error bars in tables 1, 2, 3 mean?

7. The error bars in tables 1, 2, and 3 represent the standard deviation of the steps over multiple runs of the experiment. They provide a measure of the variability of the results and allow for a more accurate comparison of the performance of our method with that of the baseline methods.

We hope the above response addresses your concerns. If you find our revisions and responses helpful, we would greatly appreciate your consideration in raising the score to support our work. Please let us know if you have any further questions.

[1] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., ... & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

[2] PrismarineJS. Prismarinejs/mineflayer, 2023a. URL https://github.com/PrismarineJS/mineflayer. https://github.com/PrismarineJS/mineflayer .

[3] Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., ... & Gui, T. (2023). The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864.

Questions:

How does the interpretability of the presented method relate to the interpretability of baseline methods, e.g., voyager?

1. ADAM's interpretability is rooted in its decision-making process, which relies on a causal graph, as previously discussed. Methods like VOYAGER [1] do not emphasize transparency or interpretability as strengths. Their interpretability is inherently limited because their knowledge relies heavily on the static prior knowledge of LLMs, rather than being acquired through interactive exploration and intervention-based causal discovery.

Isn't the interpretability of the method compromised by the lack of interpretability of the used LLMs?

2. ADAM's interpretability is primarily facilitated by its causal graph. The causal graph does not rely on the static prior knowledge embedded in the LLM, but rather is acquired through empirical knowledge gathered from continuous interactions with the environment, reasoning, and experimentation. The usage of LLMs in ADAM does not affect the correctness and interpretability of the causal graph obtained through our causal algorithm. Therefore, "the lack of interpretability of the used LLMs" will not affect ADAM's interpretability, which remains independent of LLMs. In contrast, methods that rely solely on LLMs are affected by their lack of transparency.

The proposed method is quite complex. The presented ablation studies seem insufficient to rigorously justify such a complicated system; so how was the method designed?

3. Each module in our proposed system serves a distinct function, forming a closed loop as shown in Figure 2(Line 122). The Interaction Module enables the agent to execute actions from the action space and processes the agent's observable information into formatted records. The Causal Model Module constructs the causal graph, the Controller Module selects actions and completes tasks in the environment, and the Perception Module provides visual observation description.

      Our ablation experiments include:

   1. The ablation of the Perception Module (Table 3, line 448)

   2. The ablation of the LLM-based CD (Table 4, Line 493) and intervention-based CD (Figure 8, Line 499) in Causal Model Module.

   3. (Table 1, Line 384) The ablation of on the role of the TM (Temporal Modeling) and SD (Subgraph Decomposition) (Line 311) used in the Causal Model Module.

     As for the other two modules, they are the backbone of embodied exploration. Ablating these modules would render the agent incapable of interacting with the environment, thereby precluding any discussion of performance.

How much inspiration was drawn from prior work?

4. We indeed drew inspiration from previous works, mainly in the development of the Controller module. The collaboration among Planner, Actor, and Memory is a well-established design in embodied agents, as outlined in the survey [3], where Brain (planning ability and memory), Perception, and Action form the general workflow of these agents. This design corresponds to the structure of our Controller module. Our novel contribution lies in enhancing this design to better accommodate the construction and utilization of causal graphs, as demonstrated in our experiments (Figure 1, Line 68). We hope that the clarifications provided above, the explanation of our module design's rationale, and the supplementary code materials will lead to a re-evaluation of our paper's soundness. Please feel free to ask if you have any further questions.

Adam is claimed to be a "generalizable framework"; why not back this claim with a complementary application in another environment?

5. ADAM can be generalized, and we have provided a detailed process for generalizing our approach to other domains in "Appendix F, GENERALIZATION (L 961)". Our initial intention was to use Minecraft as one of the most representative and challenging environments, with the detailed explanation in this scenario serving as a typical example for broader transfer. We have revised the relevant statement in the updated version for clarity.

Responses to Reviewer yx6R

Thank you for your thoughtful review and constructive suggestions. We are delighted for your recognition of the novelty of our method and the clarity of our writing. We provide detailed replies to your comments and hope we can resolve your major concerns.

Concerns on Weaknesses:

unsupported claims: their agent "closely aligns with human gameplay;"

1. We believe there may be a misunderstanding. Our point is that ADAM's observation space 'closely aligns with human gameplay' without the use of metadata, not that ADAM's gameplay level matches that of human players. Please refer to Line 103: "We tackle the limitations of existing embodied agents. Our ADAM achieves excellent generalization capability without relying on prior knowledge or omniscient metadata like other LLM-based agents, while exhibits human-like exploration as a general framework." Unlike methods such as VOYAGER [1], which use metadata inaccessible to human players to guide decision-making, ADAM employs a visual approach, making its process more similar to that of human players. We have modified the relevant descriptions in the updated version to clarify it.

unsupported claims: "excellent interpretability"

2. ADAM's interpretability is based on its explicit representation of knowledge and decision-making through a causal graph. This approach differs fundamentally from the inherent knowledge of LLMs or RL agents, which often lack transparent and interpretable representations.

   
   The causal graph enables us to evaluate potential errors or biases in the agent's learned knowledge and provides a clear visualization of the decision-making process. This systematic modeling of world knowledge is a novel contribution, as previous works typically rely on learned neural network weights or rule-based knowledge bases. By explicitly representing the causal relationships between world entities, our approach facilitates a more transparent and interpretable understanding of the agent.

a runtime/memory analysis (be it theoretical or empirical) is completely missing,

3. I understand your concern. All of our comparisons use the same LLM API (GPT-4-turbo), and computations are performed externally rather than locally. For the overhead of a single step, all our performance experiments are conducted on Mineflayer [2], ensuring that the definition and calculation metrics of steps are consistent across all baselines, including the number of API calls. We follow the evaluation metrics consistent with VOYAGER and other Minecraft agents.

insufficient reproducibility due to missing source code

4. We provide our source code on an anonymous website to meet the requirements of double-blind review. https://anonymous.4open.science/r/ADAM_anonymous2-B0C3

Responses to Reviewer b8Jp

Thank you for your thoughtful review and recognition of our contributions. We are glad that you found our incorporation of causal discovery methods in a modular framework to be novel and effective. We would like to address your concerns and questions.

Concerns on Weaknesses:

One concern is whether ADAM scales with more complex world and causal graph for intervention-based causal discovery (CD).

1. This is possible. We have provided a detailed process for generalizing our approach to other domains in "Appendix F, GENERALIZATION (L 961)". We briefly outline the steps here:
   1. Identify key elements related to task objectives and represent them as causal graph nodes.
   2. If additional observational data is available (e.g., LiDAR), the perception module can potentially be enhanced beyond simply relying on visual input.
   3. Discretize the continuous action space into a finite set of causal graph nodes, with granularity tailored to the specific environment.

It would be good to have at least one multi-modal baseline, e.g. Jarvis-1 [1] as this is also cited by the authors.

2. Most current MLLM agent works explicitly utilize prior knowledge, and thus there is no baseline that can be fairly compared to ADAM. For instance, Jarvis-1 [1], as referenced, explicitly uses recipe knowledge (the core tech tree) as part of its agent system (see https://github.com/CraftJarvis/JARVIS-1/tree/main/jarvis/assets/recipes). This is exactly what ADAM does not rely on**, as it learns entirely** from scratch. Without this predefined knowledge, Jarvis-1 would fail to function completely, making a direct comparison with ADAM infeasible. This further underscores the superiority of ADAM.

We hope the above response addresses your concerns. If you find our revisions and responses helpful, we would greatly appreciate your consideration in raising the score to support our work. Please let us know if you have any further questions.

[1] Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., ... & Liang, Y. (2023). Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997.

[1] Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440.

[2] Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.Advances in Neural Information Processing Systems, 35:18343–18362, 2022.

[3] PrismarineJS. Prismarinejs/mineflayer, 2023a. URL https://github.com/PrismarineJS/mineflayer.

[4] Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., ... & Clune, J. (2022). Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35, 24639-24654.

[5] Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104.

[6] Lifshitz, S., Paster, K., Chan, H., Ba, J., & McIlraith, S. (2024). Steve-1: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36.

[7] Cai, S., Zhang, B., Wang, Z., Ma, X., Liu, A., & Liang, Y. (2023). Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235.

[8] Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., ... & Liang, Y. (2023). Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997.

[9] Wang, Z., Cai, S., Mu, Z., Lin, H., Zhang, C., Liu, X., ... & Liang, Y. (2024). OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents. arXiv preprint arXiv:2407.00114.

[10] Li, Z., Xie, Y., Shao, R., Chen, G., Jiang, D., & Nie, L. (2024). Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. arXiv preprint arXiv:2408.03615.

[11] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., & Liang, Y. (2023). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560.

[12] Qin, Y., Zhou, E., Liu, Q., Yin, Z., Sheng, L., Zhang, R., ... & Shao, J. (2024, June). Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 16307-16316). IEEE.

[13] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., ... & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

[14] Hu, X., Zhang, R., Tang, K., Guo, J., Yi, Q., Chen, R., ... & Chen, Y. (2022). Causality-driven hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35, 20064-20076.

[15] BAAI, P. (2023). Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563.

Questions:

Would the method generalize to other environments? If so, what are the assumptions and requirements for the application of the method to a new environment?

1. We have provided a detailed process for generalizing our approach to other domains in "Appendix F, GENERALIZATION (L 961)". See the above response.

How does the method compare to approaches trained with reinforcement learning?

2. Thank you for the good question. RL agents and LLM-based agents focus on addressing different challenges. RL is well-suited for learning the execution of specific actions, such as gathering wood or mining minerals, but struggles with long-term dependencies. SOTA RL systems often employ hierarchical designs (e.g., CDHRL [14]), which require substantial human involvement. Effective RL models typically require (often) millions of learning steps, and their knowledge is embedded within network weights, making it difficult to express in a structured form (e.g., a graph). This black-box nature limits controllability, transparency, and interpretability. In contrast, LLM-based agents focus on addressing long-term problems, with some also integrating RL models as execution units (Plan4MC [15] , Jarvis-1 [8], etc.).

   
   One of the major contributions of our work is to alleviate the reliance on prior knowledge in LLM agents. Current Minecraft agents use LLMs that have been well-trained on the game's knowledge to complete related tasks, which has introduced bias. To address this, ADAM explores Minecraft from scratch and stores knowledge in the form of causal graphs, which enhances robustness and interpretability.

Is it possible to train an agent with reinforcement learning on the same action space that ADAM uses?

3. Theoretically, training an RL agent in our environment is feasible. However, using our commercial Minecraft to support large-scale data sampling would be costly, as each step takes approximately 2 minutes. With an action space of 41 actions, the permutations and combinations required for RL to find a successful decision path would necessitate significantly more steps, making it impractical.

What is the captioning performance of the perception module? What are its failure cases?

4. See my response 4. MLLM serves only an auxiliary role and is not well-suited for tasks requiring fine-grained positional accuracy (e.g., building a house). While using metadata could improve this aspect, our goal is to develop agents without relying on metadata.

We hope the above response addresses your concerns. If you find our revisions and responses helpful, we would greatly appreciate your consideration in raising the score to support our work. Please let us know if you have any further questions.

Responses to Reviewer Gehh

We are very grateful for your in-depth and thoughtful comments. We appreciate your recognition of the novelty and performance of our method. We are here to provide detailed replies to your comments and hope we can resolve your major concerns.

Concerns on Weaknesses:

The presentation of the method is quite high-level and does not help the reader understand how the method actually works in practice. When the different "modules" are introduced, it is not clear a priori what they actually are.

1. We respectively disagree with your assessment. Our presentation is not "quite high-level", but rather detailed and thorough. We provided an in-depth introduction to each module's function, including specific examples to illustrate how they are working. Furthermore, we used detailed diagrams to explain how each module works and how they interact (Interaction Module: Figure 3; Causal model module: Figure 4 and Figure 5; Controller module: Figure 6). Notably, all the other three reviewers have given positive evaluations regarding the clarity of our presentation, and we believe it effectively conveys the intended details. If you have any further questions, please feel free to ask, and we will do our best to clarify.

   
   Taking the Interaction Module as an example, Figure 3 provides a step-by-step illustration of the process. Starting from the initial action space {gatherWoodLog}, the figure demonstrates how the action space is sampled and how the sampled data is subsequently processed into a record. Similar detailed examples are provided for other modules to ensure a comprehensive understanding.

Are they just prompts and specifications to a GPT4 model? If so, it could be beneficial to show one of the prompts earlier, to guide the understanding of the rest of the paper.

2. The modules of ADAM extend beyond prompt engineering; they are components designed to interact with the environment in an embodied manner, as well as to extract, validate, and organize causal knowledge. Fundamentally, they rely on logical execution and the mathematical representation of environmental knowledge, rather than merely using LLMs with designed prompts. Notably, modules such as the Interaction Module and the intervention-based causal discovery (CD) in the Causal Model Module do not utilize LLMs at all.

   
   For example, the Interaction Module can determine how to sample the action space based on the task and the causal graph learned by other modules, and can parse the data obtained from the environment into formatted data for other modules. Detailed examples illustrating the operational logic of each module are provided (see above), with similar approaches applied in the other modules.

The comparisons in the paper are unclear, due to the choice of a specific action space that is different from the one used in previous work. Indeed, while the paper mostly discusses the "observation space" difference compared to the setting usually employed in reinforcement learning papers, one crucial difference is the one in action space.

3. Thank you for your suggestion. We provide a detailed description of the differences between various Minecraft agents in the appendix (L 875, Appendix C AGENT IN MINECRAFT).

   
   Here, we elaborate on the action space of current Minecraft agents. The action space is primarily determined by the environment in which the agent is deployed. Currently, there are three main types of 3D Minecraft environments used in agent research: MineRL [1], MineDOJO [2], and Mineflayer [3].

   1. MineRL [1] is represented by work such as VPT [4], DreamerV3 [5], STEVE-1 [6], GROOT [7], Jarvis-1 [8], OmniJARVIS [9], and Optimus-1 [10]. MineRL utilizes low-level actions based on keyboard and mouse operations, with some methods introducing high-level actions, such as Optimus-1 [10]'s addition of craft and smelt actions.

   2. MineDOJO [2] is represented by methods such as MineDOJO [2] (low-level action), DEPS [11] (discrete action code), and MP5 [12] (discrete action code).

   3. Mineflayer [3], on the other hand, does not provide a Minecraft environment itself, but rather requires the combination of a commercial Minecraft, thereby offering the most complex environment that has features identical to which human players experienced. Representative work using Mineflayer includes VOYAGER [13] (JavaScript code) and our ADAM (discrete action code).

It could be surprising to observe that an off-the-shelf open model is accurately describe an observation to the level of providing enough information for an actor to take the optimal action, especially in an environment that is as visually rich as Minecraft.

4. Open-source MLLMs are currently insufficiently accurate to provide detailed information for complex tasks, as discussed in my response 7 to Reviewer GNQs. Existing MLLMs primarily assist in observing peripheral environments, which has been validated through our ablation experiments. However, human feedback remains necessary for Minecraft tasks requiring high accuracy, as also noted by VOYAGER [13]. To our knowledge, relying solely on vision (without metadata) for complex tasks like building remains unfeasible, and we are committed to advancing in this direction, including modeling causal relationships in such tasks. ADAM's decision-making process relies on its memory, the causal graph it constructs**, and the reasoning ability of LLMs****,** with MLLM playing only a supporting role.

The method seems to be highly Minecraft-specific, and the paper does not extensively discuss how it could be generalized to other domains.

5. We have provided a detailed process for generalizing our approach to other domains in "Appendix F, GENERALIZATION (L 961)". We briefly outline the steps here:
   1. Identify key elements related to task objectives and represent them as causal graph nodes.
   2. If additional observational data is available (e.g., LiDAR), the perception module can potentially be enhanced beyond simply relying on visual input.
   3. Discretize the continuous action space into a finite set of causal graph nodes, with granularity tailored to the specific environment.

[1] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., ... & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

[2] Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., ... & Liang, Y. (2023). Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997.

[3] Wang, Z., Cai, S., Mu, Z., Lin, H., Zhang, C., Liu, X., ... & Liang, Y. (2024). OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents. arXiv preprint arXiv:2407.00114.

[4] Tang, X., Li, J., Liang, Y., Zhu, S. C., Zhang, M., & Zheng, Z. (2024). Mars: Situated Inductive Reasoning in an Open-World Environment. arXiv preprint arXiv:2410.08126.

[5] Hafner, D. (2021). Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780.

[6] Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., ... & Tsing, R. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782.

[7] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., ... & Whiteson, S. (2019). The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043.

Experimentally, however, it does not show significant advantages over Voyager, as it does not complete tasks that Voyager was unable to.

5. Our method has demonstrated significant advantages over VOYAGER in both the original (2.2× speedup, Line 440) and modified (4.6× speedup Line 68) environments. More importantly**, ADAM successfully obtains diamonds in the Modified environment, a feat that VOYAGER is unable to accomplish (Figure 1, Line 68).** You may have misinterpreted our ablation experiments (Line 449) and performance experiments (Line 440).

The causal graph generated by the model ADAM is quite similar to the hybrid knowledge graph in memory described in Optimus-1

6. We wish to clarify that the game rules (tech tree) of Minecraft are publicly available through the Minecraft community and various wikis, and are not exclusive to academic research. You can verify this through image search engines. This is similar for all Minecraft-related works, provided they do not manually alter the game rules. Analogously, most StarCraft agent research also relies on the standard tech tree [6][7], allowing agents to compete with human players under the same conditions.

   
   The key point is how the agent obtains this rule, which brings us back to the discussion in (2). Since the Minecraft tech tree has appeared too many times on the internet, previous LLM agents directly memorized this graph, while ADAM discovers it from scratch. We tested the baselines' performance in a modified Minecraft environment and found that their performance immediately drops significantly (Line 1055 Table 7). ADAM solves this problem by discovering causal relationships from scratch to adapt to any changing environment.

The current causal graph is entirely object-centric. In open-world Minecraft, there are many open-ended tasks, such as building and farming, which are not strictly object-centric.

7. This limitation exists in current methods when metadata is not used. Descriptions generated by MLLMs struggle to precisely indicate the relative positions of items, posing challenges for tasks requiring fine-grained positional accuracy (e.g., building a house). While using metadata could potentially improve this, our goal is to develop agents without relying on metadata, recognizing that such information may not be accessible in real-world scenarios. We hope that future advancements in MLLMs will enhance their ability to describe relative positions, an effort that will likely require contributions from the broader community.

Several relevant works are not cited, including:

[1] Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

[2] Mars: Situated Inductive Reasoning in an Open-World Environment, NeurIPS 2024

[3] OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents, NeurIPS 2024

8. Thank you for reminding us, and we have added references to these papers in the updated version.

We hope the above response addresses your concerns. If you find our revisions and responses helpful, we would greatly appreciate your consideration in raising the score to support our work. Please let us know if you have any further questions.

Responses to Reviewer GNQs

Thank you for your thoughtful review and constructive suggestions. We are delighted by your recognition of our paper's clear presentation and ADAM’s performance. We provide detailed replies to your comments and hope we can resolve your major concerns.

Concerns on Weaknesses:

It contains numerous typos and minor errors. I recommend that the authors carefully review and correct these issues.

1. Thank you for your suggestion. We believe the phrase "numerous typos" may overstate the issue, as we have not identified many. Nonetheless, we will continue to review our paper carefully. Should you notice any additional typos, we welcome your input for further improvement.

The major issue lies in the extensive use of pretrained language models that already incorporate substantial knowledge of Minecraft. Since language models may internally form a comprehensive causal graph of the game world, primarily in linguistic form, the proposed additional causal graph construction might be redundant.

2. We respectfully disagree and believe there may be a misunderstanding of our fundamental experiment. Your statement appears contrary to the findings presented in our paper. ADAM's robustness in the modified environment stems from its ability to construct causal graphs without relying on prior knowledge (Fig. 1 (Line 68), Robustness (Line 428)). Specifically, ADAM uses the reasoning ability of LLM (Line 268) and intervention-based causal method (Line 280) to obtain the causal graph. This has been clearly demonstrated in our experiments and was highlighted by Reviewer b8Jp as strengths 1 and 2.

   
   The issue you raised is actually a limitation of existing methods (such as VOYAGER [1], Jarvis-1 [2], OmniJARVIS [3]), and represents a fundamental difference between our approach and others. In the modified environment, the game rules (causal graph) of Minecraft are altered, making prior knowledge even harmful. Therefore, in such a modified environment, "LLMs incorporated substantial knowledge of Minecraft" is actually mismatched with the environment, and only constructing causal graphs from scratch allows for successful task completion. This is one of our most important experimental results (Fig. 1 (Line 68), Table 7 (Line 1055)), demonstrating ADAM's superior performance -- being the only approach capable of mining diamonds.

   
   The causal graph constructed by ADAM is not derived from the prior knowledge embedded in LLMs; instead, it emerges from an iterative intervention and reasoning process enabled by the Interaction Module and the Causal Model Module. This distinction is crucial**: prior knowledge from LLMs is inherently limited to static information, whereas ADAM's approach allows for adaptability in dynamic environments.** The learned causal graph provides ADAM with enhanced robustness to environmental changes, as demonstrated in our experiments (Robustness, Line 428).

I suggest that the authors explore scenarios with completely altered world rules in Minecraft to test the validity of models like GPT in such modified environments, perhaps using a setting like “Mars.”

3. Our robustness experiment (Line 428) has already altered the core dependencies of the game, and in the ablation experiment, we verified that the core of task performance is the reasoning ability of LLM, not prior knowledge.
   
   Mars [4] uses Crafter [5], which simplifies the environment while retaining Minecraft's prior knowledge. In other words, Mars [4] (Crafter [5]) does not "(fully) alter world rules".

The agent’s modular design is nearly identical to Voyager, with the primary addition being the causal graph.

4. We respectfully disagree. The three key components of VOYAGER [1] -- Automatic Curriculum, Iterative Prompting Mechanism, and Skill Library -- are fundamentally different from our approach. We are unclear about the basis for your conclusion, as all our innovative designs are tailored to the process of discovering (Line 205, 268), verifying (Line 280), and utilizing (Line 316) causal graphs from scratch, which VOYAGER does not address.

   
   While VOYAGER manages existing knowledge and skills via a skill library, ADAM employs causal graphs, representing a significant difference in knowledge discovery, storage, and utilization. Incorporating causal structures into embodied intelligence requires considerable effort, which constitutes our original contribution, as acknowledged by reviewers Gehh and b8Jp.

[1] Hafner, D. (2021). Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780.

[2] Tang, X., Li, J., Liang, Y., Zhu, S. C., Zhang, M., & Zheng, Z. (2024). Mars: Situated Inductive Reasoning in an Open-World Environment. arXiv preprint arXiv:2410.08126.

[3] Wu, Y., Min, S. Y., Prabhumoye, S., Bisk, Y., Salakhutdinov, R. R., Azaria, A., ... & Li, Y. (2024). Spring: Studying papers and reasoning to play games. Advances in Neural Information Processing Systems, 36.

[4] Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104.

[5] BAAI, P. (2023). Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563.

[6] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X. S., & Liang, Y. (2024). Describe, explain, plan and select: interactive planning with LLMs enables open-world multi-task agents. Advances in Neural Information Processing Systems, 36.

[7] Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., ... & Liang, Y. (2023). Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997.

[8] Wang, Z., Cai, S., Mu, Z., Lin, H., Zhang, C., Liu, X., ... & Liang, Y. (2024). OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents. arXiv preprint arXiv:2407.00114.

[9] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., ... & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.

[10] Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440.

[11] Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.Advances in Neural Information Processing Systems, 35:18343–18362, 2022.