Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots

We thank reviewer HoUA for the insightful feedback. The questions and concerns are addressed below.

It is worth considering whether additional sensor types, such as LiDAR and sound sensors, can be integrated in the future to enhance obstacle avoidance and navigation capabilities.

Thank you for your suggestion. We agree that incorporating these sensors is useful, and it is on our roadmap. Since our work builds upon the Habitat platform, it already provides support for some of these sensors (for example, sound in [1]) and it would not be difficult to integrate them into future tasks.

Fixed hand movements can lead to a decrease in visual accuracy. It may be considered to address the deformability of the skin during the simulation process and create hand motion animations during activities like grasping and walking.

Thank you for your comment. While we did not model hand movements in our work, we use the SMPL-X body format to represent the humanoids, which allows us to specify hand and finger positions. More realistic hand movement could be addressed by recording hand keypoints in motion capture data or by using generative models for hand poses [2].

This simulator has a wide range of potential applications and can be further explored to implement other embedded artificial intelligence tasks, such as visual-language navigation.

Thank you for highlighting the potential of the simulator for a wide range of applications. Given that Habitat 3.0 is based on the Habitat platform, it automatically supports well-studied Embodied AI tasks [3,4,5]. Furthermore, it opens the opportunity to update many of the existing tasks to include humanoids (e.g., “walk to the human as soon as they sit on the chair”, for visual-language navigation) or to evaluate them using our human-in-the-loop tool.

References.

[1] Chen, C., Schissler, C., Garg, S., Kobernik, P., Clegg, A., Calamia, P., ... & Grauman, K. (2022). Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems (NeurIPS)

[2] Romero, J., Tzionas, D., & Black, M. J. (2017). Embodied hands. ACM Transactions on Graphics, 36(6).

[3] Krantz, J., Lee, S., Malik, J., Batra, D., & Chaplot, D. S. (2022). Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances. arXiv preprint arXiv:2211.15876.

[4] Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., ... & Batra, D. (2021). Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems (NeurIPS).

[5] Chaplot, D. S., Gandhi, D. P., Gupta, A., & Salakhutdinov, R. R. (2020). Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems (NeurIPS).

Robotic agent diversity was not addressed.

Habitat 3.0 already supports a range of robots (Stretch, Spot, Franka, Fetch) and has the ability to include new robot models (via URDF files) since it is based on the Habitat platform.

Task diversity.

Since our work builds upon the Habitat platform, it automatically supports social variants of a wide range of Embodied AI tasks, including Image [1], Object [2] or Language [3] based navigation and rearrangement [4], or learning scene representations with embodied agents [5]. In this work, we propose two example tasks that build on navigation and rearrangement, highlighting challenges in human-robot collaboration. However, Habitat 3.0 opens the possibility to define a wide range of tasks, or even update currently studied tasks with humanoid simulation and human-in-the loop interaction. Some ways in which the tasks could be adapted include adding humanoids with complex motions (such as the ones shown in the supplementary video), asking agents to answer questions about the humanoid behavior, anticipating when a humanoid would fall, and performing manipulation tasks in close proximity or jointly with the simulated humanoids such as performing object handovers, or jointly carrying large objects.

I find the HITL evaluations more interesting, however, the paper does not cover detailed evaluation and analysis of these experiments.

Appendix G provides more analysis of the HITL experiments. Specifically, we provide post-hoc pairwise comparisons and significance results between solo execution and the two methods that we evaluate in HITL experiments.

The agent first rigidly rotates to face the next target waypoint and then follows a straight path. What are possible solutions/integrations to alleviate this?

Two possible solutions to alleviate the simplifying assumption made in our work would be: (1) turning the humanoid before reaching the waypoint in combination with a forward motion, resulting in a smoother curve around the waypoint or (2) training a motion generation model conditioned on waypoints using motion captured data, such as [8].

It is unclear how easy and flexible it is to import motion capture data. Can you elaborate on that?

Habitat 3.0 represents human motion as a sequence of joint rotations, using the same joints as in the SMPL-X body format. We provide a script to process a SMPL-X pose to be played in Habitat 3.0. As such, any motion capture data can be imported into Habitat by converting it into the SMPL-X format (using open-source tools such as MoSh++ [12]). The script to convert from SMPL-X to the Habitat format can be found in the link below, and will be open sourced. https://anonymous.4open.science/r/habitat-lab-C5B4/habitat-lab/habitat/utils/convert_smplx_to_habitat.py

It is also unclear how trivial it is to use the AMASS dataset along with VPoser to compute humanoid poses and then to import them into the simulator.

As mentioned above, we provide a script to convert SMPL-X poses into the Habitat 3.0 format. Since VPoser outputs poses in the SMPL-X format, these can directly be played by running the given script, and since AMASS has been processed in the SMPL-X format, it can be directly played as well.

We thank reviewer ndxG for the insightful feedback. The questions and concerns are addressed below. The references are in the last part of the response.

The movement of the agents lacks physical realism, which hinders the extent of how human-robot interaction can be evaluated accurately.

Habitat 3.0 supports a wide range of humanoid motion, with motion coming from learning-based motion generation models, MoCap data, or by simulating the humanoid dynamically using the Bullet physics backend, such that it responds to forces, gravity, obstacles or momentum. In this work, we choose to simulate the humanoid with some assumptions for efficient simulation, which still achieve a higher level of realism compared to the current Embodied AI simulators with humanoid support, such as VirtualHome [9] or Co-Gail [10].

However, there is another deeper question in the reviewer’s comment – how accurate does the simulator need to be? No simulation is perfect, and it is an open question how much physical realism is needed for sim2real transfer. Prior work [11] has shown that dynamic simulation and high fidelity physics might not necessarily lead to better sim2real transfer in navigation. We believe that an analogous systematic sim2real evaluation is necessary for Social Navigation and Social Rearrangement in the context of humanoid simulation to judge the level of realism needed. This is on our roadmap for future work.

The results indicate that none of the baselines achieve a useful following rate (F) nor feasible collision rate (CR) in the social navigation task.

We believe that only focusing on F alone does not provide an accurate picture of our results. First, note that our RL baseline can almost perfectly locate a moving human in an unseen house, with a success rate of 97%. The Heuristic Expert-designed baseline (which has access to a privileged map of the environment and a path planner), obtains a follow rate of 0.51 and collision rate of 0.52, showing that this is a challenging task, deserving further study. In comparison, the RL baseline achieves a following rate of 0.44, and collision rate of 0.51, without any privileged information. Finally, the following metric (F) is highly influenced by the time it takes to find the human (p). This shows that, while there is room for improvement, our baseline is able to fairly accurately follow the human around cluttered environments. Our experiments establish that these are indeed tasks where state-of-the-art approaches suffer at achieving competitive performance, further enhancing the significance of our work, and opening venues for future research.

For the social rearrangement task, none of the methods seem to generalize and let the robots assist their partners effectively (checking the relative efficiency (RE) metric). Even for HITL evaluations, which would be simpler since humans adjust to robots on the fly, the results are not encouraging.

While there is room for improvement in our current results, our baselines actually show improvement over performing the task alone. Two agents can, at most, have a relative efficiency of 200%, which would occur if both perfectly split the tasks and did not need to avoid each other (quite unlikely in a typical scenario). Therefore, in both our automated and HITL results, where the relative efficiency is above 100%, our agents effectively assist humans (a maximum of 159.2% and 109.75% for seen and unseen partners, respectively, and 133.80% in the HITL setting). While we agree that human adaptation should further enhance task efficiency in the HITL results, humans also take time to infer their partner's actions, resulting in idle steps that reduce the overall efficiency. The drop in RE for unseen partners further underlines the challenging nature of our tasks, where state-of-the-art approaches have room for improvement.

It feels like (looking at the efficiency improvements (RE metric) when collaborated vs. solo cases maybe the tasks do not offer enough opportunities for collaboration.

RE is not necessarily a measure of how collaborative a task is since it also measures the ability of a particular approach at improving efficiency over solo execution. In fact, some of our baselines such as Learn-Single (where the agent is trained with a single humanoid policy) offer non-trivial efficiency improvement (160%) when paired with seen partners. This shows that the current social rearrangement task does offer opportunities for collaboration, and merits further study in future work, especially when paired against unseen partners.

Clarification for Section 3.1. Are all the objects simplified as a point particle? Are all the objects in the environment interactive? Adding properties to objects.

Objects are not represented as point particles; instead, they have volumetric collision meshes and physical properties, which can be edited by researchers. Some objects can also be articulated allowing agents to open doors or cabinets. For the experiments in this work, we do not simulate dynamics (forces, momentum, friction, etc.) and kinematically attach the object volume to the agent arm, but the simulator allows for dynamic simulation. We have clarified this in the updated revision.

It would be beneficial to discuss how future works can incorporate more motions with interactive objects and form meaningful tasks.

Our work provides an interface to represent human poses or motions via the SMPL-X format (i.e. as a sequence of rotations for each human joint). As such, any model or system that can generate these poses, could also be used in Habitat 3.0. Researchers could for example use [4], to generate human motion from language descriptions, allowing for tasks that require robots to reason about these motions (for instance, wait for a person to sit in the chair before bringing them a glass of water).

I wonder if the HITL tools would also be standardized and open-source.

Yes, we will open-source the HITL tool, together with the simulator and experiment code. The source code for the HITL tool can be found in the following anonymous link: https://anonymous.4open.science/r/habitat-lab-C5B4/README.md.

References.

[1] Truong, J., Chernova, S., & Batra, D. (2021). Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. IEEE Robotics and Automation Letters, 6(2), 2634-2641.

[2] Yokoyama, N., Clegg, A. W., Undersander, E., Ha, S., Batra, D., & Rai, A. (2023). Adaptive Skill Coordination for Robotic Mobile Manipulation. arXiv e-prints.

[3] Truong, J., Rudolph, M., Yokoyama, N. H., Chernova, S., Batra, D., & Rai, A. (2023). Rethinking sim2real: Lower fidelity simulation leads to higher sim2real transfer in navigation. In Conference on Robot Learning (CoRL).

[4] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., & Bermano, A. H. (2022). Human Motion Diffusion Model. In International Conference on Learning Representations (ICLR).

[5] Luo, W., Sun, P., Zhong, F., Liu, W., Zhang, T., & Wang, Y. (2018). End-to-end active object tracking via reinforcement learning. In International Conference on Machine Learning (ICML).

[6] Szot, A., Jain, U., Batra, D., Kira, Z., Desai, R., & Rai, A. (2023). Adaptive Coordination in Social Embodied Rearrangement. In International Conference on Machine Learning (ICML).

[7] Chaplot, D. S., Gandhi, D. P., Gupta, A., & Salakhutdinov, R. R. (2020). Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems (NeurIPS).

[8] Krantz, J., Lee, S., Malik, J., Batra, D., & Chaplot, D. S. (2022). Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances. arXiv preprint arXiv:2211.15876.

[9] Krantz, J., Wijmans, E., Majumdar, A., Batra, D., & Lee, S. (2020). Beyond the nav-graph: Vision-and-language navigation in continuous environments. In European Conference on Computer Vision (ECCV).

[10] Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018). Embodied question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

We thank reviewer FYWe for the insightful feedback. The questions and concerns are addressed below. The references are in the last part of the response.

Universality in terms of (1) physics simulation, (2) sim2real deployment, (3) fine-grained object interaction.

Note that Habitat 3.0 is built on top of the Habitat platform, providing support for physics simulation, a wide range of tasks and examples of sim2real transfer. We provide more details below:

• Physics simulation: Habitat 3.0 supports physics simulation via bullet, allowing to specify physical properties for every object or agent and to simulate rigid-body dynamics (forces, collisions, etc). Our current work primarily uses a ‘kinematic’ simulation, which does not model dynamics. Our future work will include more sophisticated physical simulation for human-robot interaction, such as joint object manipulation and handovers.

• Sim2Real deployment: Previous works leveraging the Habitat platform have established a track record of sim2real deployments in the context of navigation [1] and mobile manipulation [2]. Since Habitat 3.0 uses the same physics and rendering engine, and all policies in our submission use depth-sensors - which have shown to have low sim2real gap [1,3] - we believe that there is potential for sim2real deployment in the tasks we study. We leave this as a venue for future work.

• Fine Grained Object Interaction: In this work, our focus was primarily on high-level interactions such as task division and reducing space interference with the other agent. In our future work, we will consider more complex interactions such as lying and sitting for our tasks. However, the inclusion of features such as dexterous manipulation remains outside the current scope of Habitat 3.0. We believe other simulators and physics engines (e.g., IsaacSim) are particularly well-suited and designed for these tasks.

Universality in terms of tasks.

We propose social variants of two commonly studied tasks in Embodied AI simulators (i.e. navigation and rearrangement), adapting them to human-robot collaboration scenarios. Similarly, we can create social variants of other embodied AI tasks, such as embodied question answering [10], object [7], image [8] or language navigation/rearrangement [9] by incorporating the humanoids. Furthermore, the simulator can be potentially used for other human-centered tasks, such as long-term scene-dependent motion generation with language input, anticipation of human actions, and human intent inference since it enables generating motions for humanoids and it includes the human-in-the-loop infrastructure to evaluate the tasks with real people.

For social rearrangement, what is the motivation for using population-based approaches?

The goal in social rearrangement is to build robot agents that can rearrange objects together with humans, who can perform this task in different ways. An effective robot agent needs to adapt to its partner’s preferences to effectively assist them. We frame this as a zero-shot coordination problem and refer to [6], which has demonstrated that training agents with a diverse population enables this type of adaptation. Note that we also present a baseline (Learn-Single) which does not use a population, and instead learns to coordinate with a single partner. Our experiments show that this baseline performs worse than population-based approaches at zero-shot coordination with unseen partners (in both automated and human-in-the-loop settings).

Relationship between Active Object Tracking and Social Navigation.

Thank you for the reference; we have updated the paper with the relationship of our task with Luo et al. [5]. We note two important differences between [5] and the SocialNav task: First, in [5] there is no robot to control, and thus, does not consider important human-robot interaction aspects that we observe in our work, such as being aware of the robot’s physical extent, avoiding human-robot collisions by maintaining a safe distance or backing up when necessary to unblock the human. Second, [5] assumes that the human to track is mostly in the field of view of the robot, whereas in our case, the robot starts far away from the human and thus has to first explore the environment to find the human (Find phase) before following and tracking it around the environment (Follow phase).

Thank the authors for the detailed response, and sorry for the delay in response. My remaining questions are as follows:

1. Given that the Habitat environment is bullet-based and naturally supports physics simulation, why does this work primarily consider kinematic simulation? Is this due to the focus of current human parametric models, such as SMPL, on kinematics without incorporating physics?
2. Currently the object interactions are somewhat simple, and do not make full use of the interactive objects (e.g. doors or cabinets). These objects can make the proposed tasks (and other potential ones) more interesting. For instance, in virtualhome people can open the cabinet and search for objects. Is it because combining these object state changes with realistic human motions is more challenging?
3. Compared to existing projects like VirtualHome, this work seems to allow greater freedom in movement, such as freeform walking, but offers more limited action types and object interactions (limited to walking and pick/place actions). Given that VirtualHome is more symbolic and rule-based, I am curious if we can merge these advantages to enhance the versatility of human-centric virtual environments.

I appreciate the authors' discussion and insights on these questions.