How hard are the three environments

The HAZARD challenge's difficulty primarily comes from its constantly changing environments. We believe it is challenging enough to effectively differentiate the performance of various methods. However, it is not excessively difficult, as all agents can complete the task to some degree. Specifically, the challenge involves planning in dynamic environments, comprehending the effects and dynamics of these changes, and accurately perceiving the evolving surroundings.

Which decision-making approaches do you think will dominate in the HAZARD challenge?

Thank you for your question. Based on current results, LLM-based agents demonstrate strong decision-making capabilities, while MCTS-based agents excel among other baselines. The success of MCTS may be attributed to its natural ability to simulate environments, leading to a deeper understanding of potential future states. Therefore, we identify two key factors for superior performance in HAZARD:

• Agents capable of simulating environmental changes and imagining future states accurately during decision-making, then basing their decisions on these internal simulations.
• Agents powered by more robust foundational models. Our research reveals that LLM-based agents with a GPT-4 backbone outperform those using GPT-3.5, and GPT-3.5 surpasses Llama-13B. This trend indicates that LLM-based agents benefit from stronger foundational models.

The rendering quality and motion effects of the simulated scene are still somewhat different from reality, which may result in some perceptually based agents using camera images that will lead to domain gaps.

Thank you for your comments.

We view rendering quality as a balance with simulation speed. Given the large scale of fire and flood in the HAZARD challenge, we abandon pursuit for fine-grained simulations, such as detailed fire particle effects and fluid simulation, to evaluate embodied agents more efficiently. Additionally, our work utilizes the ThreeDWorld simulator, which comparatively offers realistic visual effects among current simulators.

The current scenario exhibits a limited scale. Is there a method available to efficiently expand the current scenario, such as loading assets and assigning attributes to different disaster scenes?

Yes. Our codebase includes documentation on efficiently adding new assets and assigning attributes within HAZARD. We have updated the documentation with more detailed instructions for loading assets and assigning attributes.

The proposed LLM pipeline has to input the same task description at each step to perform sequential decisions, which is not efficient.

We apologize for any confusion. In the chat model, the chain-of-thought reasoning's second step only requires previous conversations and an added prompt stating, “Answer with only one best next action. So the answer is option". For decisions at different time steps, it is necessary to input the task description each time, as LLM-based agents struggle to make decisions without the description in their context.

The mission objectives of the benchmarks are relatively simple, focusing on the task of object rescue, and the benchmarks can be expanded further in terms of mission complexity.

Thank you for your insightful comments. We are developing more diverse actions to enhance agents' interaction with the environment. For instance, we've introduced a 'put-out-fire' action, enabling an agent to extinguish fire at a specific location (please refer to the current ‘put out fire’ gif on our website). We are also developing actions for water pumping and wind blocking. Such additions will not only increase mission complexity but also make the challenge more engaging.

Moreover, we wish to highlight that HAZARD's main difficulty lies in its dynamic environmental changes, which is our primary focus. Considering the baseline methods' performance on HAZARD, we believe the challenge already presents sufficient difficulty.

Would there be any difference if the benchmark wasn't implemented with a nice graphics pipeline but instead as a fully text-based game a la nethack? My understanding is that at least the main LLM planner method would work without change…It would be nice though if a similar array of different options was provided on the action representation side. The current benchmark only supports very high-level actions.

Thank you for your comment. Unlike text-based games, our challenge supports the simulation of continuous observations and low-level actions, such as moving a specific distance or turning by a precise degree.

Following your suggestion, we have provided interfaces for using low-level action APIs for agents, along with relevant documentation. However, when limited to low-level actions alone, current agents struggle with task completion. For example, when we replace the action set of wind scenes with low level actions {turn, move forward, pick up, drop} instead of {walk to nearest, pickup nearest, drop}, the RL agent almost fails to retrieve even one single object.

We preliminarily test the RL agent equipped with low-level actions in wind scenes. However, as the following results demonstrate, the agent almost fails in this setting, indicating that this new setting poses significant challenges to the agents.

It is further unclear whether the RL policy that the authors compare to has all the same privileged observation and action primitive access that the LLM planner has. A fair comparison would be good here.

Yes, the observations are the same. We modify the observation to fit a 2D-array for RL and text description for other methods, but they all have access to full observation. We modified the actions of RL a bit as described in the paper, as it’s infeasible for RL to output exact object indices.

It would be nice to investigate certain axis of generalization for the different agents more systematically. E.g. one could specifically test generalization to larger rooms or to new objects etc.

In response to your valuable suggestion, we have created two new test sets: one with new objects and another with larger rooms. We've also provided documentation to facilitate the addition of new objects.

In addition, we propose a method to tackle the test set with new objects in the ‘with perception' version of HAZARD. We have replaced the R-CNN perception module with Grounded-SAM to generate segmentations for scenes containing new objects. We examine this new perception module on the test set with new objects, and results are summarized as follows:

According to the results, with the help of the strong segmentation model, all baseline models are able to retain most of their effectiveness when encountering new objects. The results are included in Appendix C.

The agent currently has no way of influencing the dynamics of the environment

Thank you for your insightful comments. We are developing more diverse actions to enhance agents' interaction with the environment. For instance, we've introduced a 'put-out-fire' action, enabling an agent to extinguish fire at a specific location (please refer to the ‘put out fire’ gif on our website). We are also developing actions for water pumping and wind blocking. Such additions will not only increase mission complexity but also make the challenge more engaging.

Is the robot agent itself incurring damages as it moves eg through fire / water?

Currently not. We value your suggestion and have accordingly integrated the impact of hazards on agents into the HAZARD challenge. To achieve this, we've developed a new setting where agents are directly affected by environmental effects:

• In fire scenes, the environment now causes damage to agents from fire by influencing the agent's temperature. This adaptation requires modifications to all baseline methods, particularly for fire scenes, as fire often blocks paths, demanding a different approach of planning algorithms.
• For flood and wind scenes, we've introduced a setting where agents are influenced by drag forces. Detailed descriptions and results of baseline methods in this new setting can be found in Appendix B. Given that most baseline methods show only a slight performance decrease, it suggests that environmental factors slightly increase the challenge in both flood and wind scenes.

Is there a way to generate training demonstrations, e.g. for an imitation learning pipeline?

We have produced demonstrations on a training set using an oracle planner with access to complete ground truth information. After obtaining the ground truth information for all time steps, the oracle planner traverses all possible rescue plans and identifies the one with the highest value. If multiple plans have the same value, the plan with the fewest time steps is selected. However, this planner assumes successful action execution (e.g., unobstructed navigation), so it may not be perfect. This part of discussion is included in Append D.

What simulation speed does the current simulator support? can it be run on headless servers & with multiple workers in parallel? This is important for understanding whether it can support RL workflows.

The HAZARD benchmark, developed on the ThreeDWorld simulator, can accommodate multiple workers on a single machine and is compatible with headless servers equipped with an X11 server. For further details, please refer to the ThreeDWorld documentation (https://github.com/threedworld-mit/tdw/tree/master/Documentation). Slow simulation is a common challenge with many simulators, but it can be mitigated through a large pool of parallel processes or by setting a lower resolution.

It seems that the hazard’s effect on the functionality of the embodied agents is not simulated.

We value your suggestion and have accordingly integrated the impact of hazards on agents into the HAZARD challenge. To achieve this, we've developed a new setting where agents are directly affected by environmental effects:

• In fire scenes, the environment now causes damage to agents from fire by influencing the agent's temperature. This adaptation requires modifications to all baseline methods, particularly for fire scenes, as fire often blocks paths, demanding a different approach of planning algorithms.
• For flood and wind scenes, we've introduced a setting where agents are influenced by drag forces. Detailed descriptions and results of baseline methods in this new setting can be found in Appendix B. Given that most baseline methods show only a slight performance decrease, it suggests that environmental factors slightly increase the challenge in both flood and wind scenes.

How would the proposed approach address a partially broken embodied agent? Would approaches similar to Zeng [1] work?

Currently, our approach to hazards can involve blocking paths or applying additional forces on agents, which do not result in partially broken embodied agents. Zeng [1]'s work examines different agent action impacts, distinct from our study of environmental change impacts on agent decisions. We have incorporated this discussion into our related work section.

The reviewer found the baselines in the experimental section somewhat weak. Comparing existing works, such as Landi [2] for embodied AI on changing environments could make the experimental section more convincing.

Following your advice, we have introduced a new baseline, MCTS-diff, building upon the previously strongest baseline, MCTS. MCTS-diff adopts the concept from Landi's work, encouraging agents to move towards areas where environmental changes are more probable. This is implemented by assigning additional value to MCTS nodes with significant temperature or flood signal changes, guiding agents to areas with rapid environmental changes.

However, Landi's work primarily focuses on identifying differences between two scenes, leading their method to motivate agents towards areas with probable environmental changes. In contrast, our environment has continuous changes. Our objective is to evaluate agent decisions under these ongoing environmental changes, rather than merely identifying the changes. As the following results indicate, MCTS-diff performs worse than MCTS. This suggests that exploring areas with environmental changes may not be an effective strategy in the HAZARD challenge.

The summarization of this discussion has been included in our related work section.

Could you elaborate why the MCTS-based method outperforms other baselines in terms of fire step and flood step?

The MCTS method outperforms other baselines in both Step and Value metrics. Compared to LLM-based agents, MCTS-based methods consistently pursue the target with the lowest cost without altering their target during navigation or replan chances. On the other hand, LLM-based agents might choose a different target upon reaching the maximum navigation steps and having the opportunity to replan. Consequently, MCTS agents typically achieve a lower Step count than LLM-based agents.