UnrealCV Zoo: Enriching Photo-realistic Virtual Worlds for Embodied AI Agents

W5: There are no language-modal input/communication among agents. The multi-agent feature is not highlighted.

R5: Language-based input and agent communication can be implemented through the gym interface without modifying the UE environment. Researchers can easily create custom multi-agent interaction tasks using this interface, e.g., cooperative navigation with communication. The primary challenge in multi-agent scenarios stems from the increasing computational and interaction costs that arise as more agents are added. As shown in Table 2, UnrealZoo demonstrates improved efficiency in supporting multi-agent interactions.

Q1: About the choice of base engine, is UE the only engine that can render photo-realistic images?

R6: While Unreal Engine (UE) is not the only engine capable of rendering photorealistic images—others include Omniverse, Unity, and Blender—UE's primary advantage lies in its open-source and mature ecosystem. Its open-source nature allows us to modify the game engine for potential future features. The Epic Marketplace/Fab provides a wide range of high-quality content created by professional developers, allowing researchers and environment developers to access diverse assets and tools that accelerate development and enhance realism in simulated environments. This enables us to scale up our collection of environments with minimal effort. Additionally, numerous developers, particularly those working on AAA video games, actively contribute to Unreal Engine, ensuring we benefit from continuous updates to the game engine, plugins, and content.

Q2: There is a lot of Agent body mentioned in Table 1. Are they driven by pre-defined policy? What are their action spaces?

R7: Except for the drone, all agents are controlled using basic movement parameters of angular velocity and linear velocity, which can be adjusted either step-wise by the user or automatically through the built-in navigation system. The drone, on the other hand, is controlled by applying forces along the x, y, and z axes, as well as yaw rotation, offering fine-grained movement capabilities. Special interactive actions, such as opening doors, picking up objects, or simulating falls, are executed through discrete control signals. We provided detailed usage APIs and video demonstrations on this website, offering further insights into the agents' control mechanisms and capabilities.

Q3: Cannot find Table 3.3 in line 308 and Table 4.1 in line 368. GPT4-o in Table 5 should be GPT-4o.

R8: We apologize for the confusion. We have corrected the table references in the paper. The correct references are now Table 2 (line 308) and Table 3 (line 368). We have also changed GPT4-o to GPT-4o. We appreciate your careful review, which has helped us improve the clarity of our paper.

Q4: Why GPT-4o performs so poorly? Do you do the error breakdown analysis? Can you use other information like position or third-person view to increase the success rate?

R9: Thanks for the suggestion. To provide clarity, we have recorded videos showing how the GPT-4o agent working on both navigation and tracking tasks. The videos can be found here. Roughly speaking, we find that GPT-4o's lack of 3D spatial reasoning with ego-centric observation and the heavy latency in inference will also make the GPT-4o agent fail in dynamic environments, e.g., tracking.

To ensure a fair comparison, we evaluated all agents using identical observation space in each task. For active tracking, agents received only first-person images, while navigation tasks provided both first-person images and the target's relative coordinates.

Q5: In the social tracking experiment, as the training set grows, the growth of SR is intuitive. Can you conduct the experiment on the same scale training (compared to the less diverse data) to give a fair evaluation?

R10: We trained the offline RL policy using the same amount of training data. Each of the three datasets (1 Envs., 2 Envs., 8 Envs.) contains exactly $100k$ samples, ensuring a fair comparison. The details of collected data are introduced in Appendix B.4.

Thank you for your valuable feedback. Below, we address each of your points in detail.

W1: There are only 100 scenes and they are not scalable.

R1: We do not agree with that at all. It is unfair to simply compare the number of scenes at different scale level. The scale level and diversity of our collected environments are significantly greater than in previous indoor environments. For example, there are numerous houses and rooms in a town-level scene, such as SuburbNeighborhoodDay. The largest scene (Medieval Nature Environment) covers more than $16 km^2$ areas. As is shown in Table 1, the scenes collected in UneralZoo are of much greater diversity in scene types, scale, and playable entities than previous works.

Regarding scalability, we provide a set of functions to manipulate the scene, including spawning new objects, destroying objects, setting object locations and rotations, and rescaling object sizes. Users can easily customize new scenes based on our pre-built environments. The illumination, camera parameters, the texture of objects, and animations of players can also be controlled by UnrealCV+. Moreover, we will also provide detailed tutorials to help users package new environments with UnrealCV+, fostering community contributions. Hence, most environments in Fab can be added to UnrealZoo with minimal effort.

W2: There is no comparison with more recent related works like [1][2].

R2: Thank you for your suggestion. In the revision, we have included two related works in Table 1 for comparison. Specifically, LEGENT[1] focuses on multi-modal interactions between agents and objects, offering only two pre-built scenes. There is only a virtual human character and a robot-like character in the scene. It is built on Unity, and the communication code has not been open-sourced yet. In contrast, Unreal Engine provides more advanced rendering, and UnrealCV has been open source for years. Our extension (UnrealCV+) will also be open-sourced upon the official public release of the environments. V-IRL[2] utilizes Google Maps APIs to allow agents to navigate in reconstructed large-scale open worlds. However, it only includes static outdoor scenes, and the agents cannot interact physically with any objects.

W3: The motivation is not clear in introduction. What's the disadvantages of previous simulators? How does UnrealZoo settle them?

R3: Our ultimate goal is to simulate complex open worlds to build generalist embodied agents for real-world applications. The main limitation of previous simulators is their lack of diversity in scenes and characters. Many focus on specific scenarios, such as daily home activities or urban autonomous driving, which hinders the advancement of generalist embodied AI in open worlds. For example, simulators like AI2-THOR, OmniGibson, VirtualHome, and Habitat primarily emphasize indoor interactions for household robots. In contrast, UnrealZoo provides a collection of high-quality, photo-realistic, large-scale scenes that encompass a variety of environments, thereby offering a more comprehensive research platform. Additionally, UnrealZoo features diverse playable entities, including humans, animals, robots, and drones, facilitating research into cross-embodiment generalization and heterogeneous multi-agent interactions.

Minecraft is a choice to simulate the open worlds but the photorealism is limited. The photorealism of the environment is required to reproduce the real-world challenges to train and evaluate the embodied visual agents for real-world applications. With advancements in Unreal Engine, UnrealZoo achieves photorealistic rendering, enhancing the realism of the simulated environments to mitigate the sim2real gaps.

W4: The interaction between agents and the world is not discussed, which is important for embodied training.

R4: We appreciate your concern regarding the diversity of interactions and actions in our environments. However, we would like to clarify that our action space already supports a broad range of interactions, such as picking up objects, climbing obstacles, opening and closing doors, entering and exiting vehicles (cars or motorbikes), falling, crouching, PTZ control, and so on. We provide video demonstrations of some typical actions on this website to demonstrate the built-in actions. These interactions enable us to simulate complex behaviors and tasks in open-world environments, showing its potential to support a wide range of embodied tasks in future research. Furthermore, Unreal Engine's flexibility allows us to customize and add new actions as community needs arise.

I didn't see GPT failure case on visual nav task in provided videos. I don't agree with GPT-4o's lack of 3D spatial reasoning. In works such as [1], LLM-based spatial reasoning can perform very well.

R3: We apologize for not providing the failure case earlier. Now, the failure case analysis has been updated on this website. The initial showcase is to demonstrate that GPT-4o is still very inefficient in planning and frequently takes some additional or wrong actions, even though it can sometimes complete this task. However, in our experiments, the number of failure cases far exceeds the number of successful ones. We have added the normal failure cases with analysis on our website to provide more context.

For 3D spatial reasoning, we utilize GPT-4o, a large vision language model (VLM), to process raw pixel 2D images and task descriptions as input and produce actions in an end-to-end manner without additional computer vision models. In contrast, Transcrib3D employs an LLM-based method that incorporates an additional 3D object detector to process colored point clouds and generate a comprehensive list of objects as text for large language models (LLMs). Therefore, this work can not demonstrate the 3D spatial reasoning capabilities of VLMs, as it relies on LLMs instead.

To clarify, we do not deny that GPT-4o is of fundamental multimodal reasoning abilities, such as identifying target categories in images and providing detailed scene descriptions. However, based on our own experiments and other recent works(Chen, et al. CVPR 2024 and Feng, et al. SIGRAPH Asia), it is clear that the 3D spatial reasoning ability of VLMs remains limited, e.g., GPT-4o or GPT-4V. They often generate hallucinated responses. Our newly updated failure demo on the website also highlights such problems. For instance, in a key moment where a left turn was needed, GPT mistakenly suggested a right turn. This led to a series of cumulative errors and ultimately resulted in the failure to accomplish the task. As validated by Chen, et al., this limitation is primarily due to the training datasets used, which typically lack spatially rich data or high-quality human annotations for 3D-aware queries, rather than the model's inherent architecture. Inspired by this work, a future application of our environments could automatically generate large-scale images and videos with detailed 3D annotations to improve the spatial reasoning capabilities of VLMs. We will include these clarifications in the revision.

Thank you for your valuable feedback. Below, we address each of your points in detail.

W1-1: License issue.

R1: We do not violate the mentioned statements (section 5.a in Fab) in eula, as we do not share the content on a standalone basis (in source format). Instead, we package all the content in an executive binary project, similar to the binaries in unrealcv model zoo. Distributing such executive binary is allowed by Epic Content License Agreement 3.a :

You may Distribute Licensed Content incorporated in object code format only as an inseparable part of a Project to end users.…...This means, for example, you may Distribute software applications (such as video games) that include Licensed Content to the general public, whether directly by you or through a distributor or publisher.

Note that all the contents in UnrealZoo were purchased from Unreal Engine Marketplace before Oct.1 (When Epic Marketplace was replaced by Fab), so we refer to the Epic Content License instead of Fab. Besides, the binary distribution is also allowed by Fab License(https://www.fab.com/eula), referring to section 4 rather than section 5. This enables us to further purchase the environments in Fab to scale up the collection in the future.

W1-2: Moreover, if the binary distribution of the environments is legal, the fact that this has to be shipped in a binary package greatly limits the open-source nature of the project.

R2: The UnrealCV command system provides a set of built-in commands to support various run-time modifications, enabling users to flexiblely manipulate content in the packaged environment. Moreover, in UnrealZoo, we further add a set of blueprint functions, which is callable by UnrealCV, to support more advanced environment customization, such as changing the textures, lighting, or weather.

Following UnrealCV, the UnrealCV+ plugin also uses the MIT License. We will open-source UnrealCV+ to help contributor package their own projects and share them with the community.

There is a trade-off between flexibility and ease of use. Although UnrealCV has been available for 8 years, most AI researchers tend to prefer using off-the-shelf binaries rather than learning Unreal Engine Editor and creating new environments from scratch. We believe that providing a large-scale collection of pre-built environments and playable entities for the AI community will help researchers bypass the costly process of building virtual worlds and explore new avenues in their field.

W2: I strongly believe that the overall presentation and writing quality of the paper should be improved.

R3: Thank you for your detailed comments. Following your suggestions, we have addressed all the listed issues in the revision. The main updates include:

• Revised the text in the related work section to focus on realistic simulators and added two recent works for comparison.
• Added the implementation details in Appendix C, covering RL algorithms, NN architectures, hyperparameters, human benchmarks, and prompts for VLM agents.
• Added the results section in Appendix D with training curves and detailed results for each environment.
• Replaced "accumulated reward" with "episodic return" in the evaluation metrics.
• Restructured the appendices:
  • A UE Environment
  • B Exemplar Tasks
  • C Implementation Details of Agents
  • D Additional Results
• Included a table in Appendix A to clarify the icons used in Table 1 and a figure comparing the visual realism of different engines.
• Added snapshots of all environments used in the experiments to the Appendix.
• Revised the reference formatting as per your suggestions.
• Fixed all minor issues.

Q1: What is the motivation behind cross-embodiment? Why is cross-embodiment generalization an interesting capability for embodied agents? Is there any practical or real-life scenario where cross-embodiment generalization is relevant?

R4: In the real world, robots typically have different mechanical structures and hardware configurations. Training separate policies for each hardware configuration is inefficient. Therefore, we need to build generalizable agents that can work across different embodiments. For example, if a tracking agent has good cross-embodiment capabilities, we can efficiently deploy one model across quadruped robots, wheeled robots, or even humanoid robots, significantly reducing the costs.

Q2: What happens when the agent's implementation can not handle the control frequency specified by the environment? Are the actions repeated or no action is taken (something like no-op in some RL environments)? Is it possible to freeze the environment to wait for the action of the agent?

R5: Without frequency control, the agent runs at 6~8 FPS, as the vision fondation model taks about 150ms at each step. Note that the results in Figure 5 and Table 5 are evaluated without frequency control. We add the new results (w/o control) in Table 4 for reference.

The frequency control is implemented by adjusting the simulator's clock based on the interaction frequency between the agent and the environment. We simulate real-world closed-loop control frequency by controlling the ratio between simulated time and engine time. This approach eliminates the need for repeated actions or no-op states to control frequency. Compared to repeated control, time dilation can maximize computational resource utilization instead of wasting time repeating actions.

You can also freeze the environment to wait for the agent's action by calling the unrealcv command vset /action/game/pause in Python.

Thank you for your thorough review and constructive feedback on our manuscript. We appreciate the time and effort you have invested in evaluating our work and are committed to addressing the concerns you have raised.

1. We emphasize that the source link for each environment is available in scene gallery. Hence, anyone needing to modify the environment from the source can directly buy and download the licensed content from the public marketplace. With the licensed content, the following developers only need to apply our open-sourced UnrealCV+ Plugins in the UE project to reproduce the environment in UnrealZoo. There is no free lunch! If the community only pursues completely free and open-source content, artists will be discouraged from contributing more high-quality content and hinder the researchers from accessing large-scale high-quality environments, ultimately slowing the advancement of the related AI fields. Therefore, we believe our distribution strategy effectively balances the trade-off between diversity of content and customization flexibility for contributors. In your initial reviews, you mentioned that the license and open-source issue is one of your two main concerns. Please be consistent with your initial comment and reevaluate your rating.

2. We agree that the writing of a paper should be standardized and rigorous, and we will continue to iterate and improve the writing of the paper for this goal. However, most of the issues you raised have been fixed or can be fixed in the final version. So we argue that these issues in the presentation should not be the main reason for rejecting this paper. The issues you recently mentioned are very minor, and they can be fixed within a few minutes. In our current version, we have:

• describe the meanings of the number in the Caption of Table 5, i.e.,"Each cell presents three metrics from left to right: Average Episodic Return (ER), Average Episode Length (EL), and Success Rate (SR)."
• In the current version, we directly refer to the related Table and Figure included in the Appendix. For example, in the caption of Table 1 (L109-L110), we mentioned Table 6 and Figure 6 (Appendix A) for convenience. In L 415, we mentioned Figure 10 (Appendix A) and Figure 14 (Appendix D). In L431, we mentioned Table 11 (Appendix D). If you prefer references to the corresponding sections in the Appendix, we will include them in the final version.
• We apologize for the oversight with Table 6 and have replaced the image with a proper LaTeX table, which is now clear and fits the academic standards of the paper.
• We apologize for the oversight of the reference issues. Note that not all the conferences have proceedings, e.g., ICLR. So it is reasonable that some conferences do not have "Proceedings". AI2-THOR has been updated, and we will further fix the other misuse capitalization issues in the final version. The journal and Arxiv papers use the @article, and the conference papers use @proceedings.

Overall, we appreciate your detailed comments to improve our writing. Thank you again for your valuable feedback. We will continuously work on this work to improve the presentation and make our contribution as valuable as possible to the community. We sincerely hope you will re-evaluate this work and provide a fair and objective score.

Thank you for your valuable feedback. Below, we address each of your points in detail.

W1: There is little diversity in the types of interactions and actions the agents can have in these environments.

R1: We appreciate your concern regarding the diversity of interactions and actions in our environments. However, we would like to clarify that our action space already supports a broad range of interactions, such as picking up objects, climbing obstacles, opening and closing doors, entering and exiting vehicles (cars or motorbikes), falling, crouching, PTZ control, and so on. We provide video demonstrations of some typical actions on this website to demonstrate the built-in actions. These interactions enable us to simulate complex behaviors and tasks in open-world environments, showing its potential to support a wide range of embodied tasks in future research. Furthermore, Unreal Engine's flexibility allows us to customize and add new actions as community needs arise.

W2&Q1: It is a little hard to understand what the "ceiling" for each task is in terms of performance. How hard are the given tasks for current RL algorithms?* How hard are the given tasks for current RL algorithms? With some limited computing, is it possible for RL algorithms to achieve the same performance as a human expert today?

R2: Thank you for your suggestions. For navigation, we have reported human performance in Table 3 as a reference. There remains a significant performance gap between RL algorithms (A3C) and human performance. For tracking, the ideal ceiling of the episode length and success rate in the current setting should be 500 and 1.00, indicating the tracker continuously following any target in any scene. The episodic reward's upper bound equals the maximum episode length, i.e., 500. At each step, if the tracker accurately places the target in its expected location, the tracker receives a reward of $+1$. Otherwise, the tracker additionally receives a penalty based on position error. We have added the details of the reward function in Appendix B. In practice, it is impossible for this reward to reach 500, as the error is inevitable. When the episodic reward reaches 400, it is already a good tracking performance.

To the best of our knowledge, the offline RL solution (Zhong et. al, ECCV 2024) is the best choice to train embodied visual tracking with limited computation resources, requiring only a consumer-level GPU for training. While current RL algorithms achieve near-optimal performance in simple environments, the main challenge lies in generalizing to diverse environments and targets. The results in Figure 5 demonstrate that RL agents can benefit from increased diversity in training environments and datasets. With the advancements in offline reinforcement learning and the increasing richness of virtual worlds, we believe achieving human-level expert strategies is within reach in the future.

W3: Inclusion of results for only DowntownWest feels lacking. Would like to see what performance looks like for other environments too.

R3: For embodied visual tracking, the results in Figure 5 were evaluated in 16 environments (4 types). In the revised appendix, we add the snapshot of the 16 environments (Figure 7 and Figure 8) and the detailed results ( Table 11) for more details. For visual navigation, we evaluate the agents in 2 environments (Roof and Factory). For social tracking, we additionally evaluate the agents in the other 4 unseen environments (Table 12). In most scenes, tracking performance declines as the number of distractors in the environment increases. The details of these new results and analysis are included in Appendix D.

Q2: How long does it take for an RL agent to become an expert in these environments? Another interesting problem to explore in RL is becoming an expert fast. If it takes unreasonably long to be.

R4: Thanks for your suggestion. We agree that exploring how to efficiently build an RL agent into an expert is an interesting problem. Thus, our experiments were conducted on a customer-level GPU (GTX 4090), which is affordable to most AI researchers. For embodied visual tracking, it takes 1 hour to collect 100k offline data and 6 hours in training to get the best tracking policy using CQL (a popular offline RL algorithm). For visual navigation, it takes 6 hours to achieve convergence in the IndustrialArea environment with four workers using A3C (a popular online RL algorithm). In the Roof environment, it takes 3 hours to achieve convergence with six workers. In the Appendix D.1, we provide the training curves. Note that applying more advanced GPUs will further reduce the training time.

We sincerely thank you for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback to confirm whether our replies have addressed your concerns. If you have any remaining questions, we are happy to provide further clarification. If our responses have resolved your concerns, we would be deeply grateful if you could consider raising the score. Thank you again for your time and effort during the review process.

W3: I am concerned about the value of the contribution.

R3: Collecting diverse photorealistic scenes at scale can simulate various real-world challenges that occur in the open world. essential for training robust agents and evaluating their generalization for real-world deployment. Note that UnrealCV has been broadly applied in a great number of pioneer works in visual intelligence for data generation, model diagnosis, and interactions, as shown in the paper list. As UnreaCV+ introduced in UnrealZoo is compatible with the UnrealCV command system, the researchers in the related areas, such as semantic understanding, 3D vision, and embodied vision, can directly access UnrealZoo and benefit from the enriched photorealistic environments, e.g., scaling data generation, diagnosis the vision models in more complex environments, or training more robust interactive policy.

Besides, with the advancement in large vision-language models (VLM), such photorealistic virtual worlds with unprecedented complexity and diversity are demanded by the community to further benchmark the VLM agents, particularly for scene understanding, dynamic modeling, social reasoning, and long-horizon decision-making.

Moreover, the toolkit (the UnrealCV+, gym interface, and wrappers) we developed for UnrealZoo can mitigate the domain gap between the game developer and the AI researchers, beneficial to both communities. For example, game developers can use these tools to effectively build AI agents as NPCs in their games, and AI researchers can more conveniently connect to the AAA game to collect high-fidelity data and benchmark the models/algorithms.

Thank you for your valuable feedback. Below, we address each of your points in detail.

W1: My biggest concern with this paper is that the proposed tasks and simulator lack a defined direction. The reviewer is unaware of any existing hardware platforms that would be sensible deployment targets.

R1: Our ultimate goal is to simulate complex open worlds to build generalist embodied agents for real-world applications. The main limitation of previous simulators is their lack of diversity in scenes and characters. Many focus on specific scenarios, such as daily home activities or urban autonomous driving, which hinders the advancement of generalist embodied AI in open worlds. UnrealZoo provides various playable entities, including humans, quadcopters, quadruped robots, cars, animals, and motorcycles. Several related works[1-5] have demonstrated the feasibility of transferring the models trained in Unreal Engine to robotic hardware platforms, including drones [1, 2], wheeled robots[3,4], and quadruped robots [5]. UnrealZoo can be used for generating high-quality ego-centric data in complex scenes, speeding up the training process, and evaluating the robustness of the learned policy before real-world deployment, showing the potential applications of UnrealZoo for robotic hardware deployment.

Currently, our primary focus is exploring fundamental challenges in embodied perception, reasoning, decision-making, and learning, with an emphasis on adaptability and generalization in scaling complex virtual worlds. Deploying the agents trained in UnrealZoo across different existing hardware platforms, such as drones, humanoid robots, and quadruped robots, in complex real-world scenarios will be our future work.

References:

[1]"Enhancing optical-flow-based control by learning visual appearance cues for flying robots." Nature Machine Intelligence

[2]"Air-M: A Visual Reality Many-Agent Reinforcement Learning Platform for Large-Scale Aerial Unmanned System." IROS 2023.

[3]"End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning." IEEE T-PAMI, 2019.

[4]"Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL." ECCV 2024.

[5]"RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking." AAAI 2023.

W2: While there are multiple criteria for these environments, one key one is speed -- the environment itself must be very performant as the ability to quickly iterate on new ideas is key. UnrealZoo is unfortunately very slow by these standards.

R2: We acknowledge the importance of speed as a criterion for evaluating simulation environments. UnrealZoo achieved comparable performance to other photorealistic environments, such as ThreeDWorld (based on Unity) and OmniGibson (based on Omniverse). Our experiments have demonstrated that the RL agents can be trained in UnrealZoo with an affordable time cost (1~6 hours) using a customer-level GPU (RTX 4090).

Fast prototyping and iteration can be achieved through various methods that do not only depend on high interaction frequencies within the environment. Techniques such as distributed training, parallel execution across multiple virtual environments, and offline reinforcement learning effectively address the interaction speed limitations of online RL. The advent of high-fidelity simulation environments has also made it possible for researchers to collect large volumes of high-quality data. Offline RL and imitation learning, in particular, leverage these pre-collected datasets for efficient policy learning, significantly reducing reliance on real-time interactions during training. We believe that as these learning approaches gain traction, high-fidelity environments like UnrealZoo will play a crucial role in supporting the development and evaluation of embodied agents.

We sincerely thank you for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback to confirm whether our replies have addressed your concerns. If you have any remaining questions, we are happy to provide further clarification. If our responses have resolved your concerns, we would be deeply grateful if you could consider raising the score. Thank you again for your time and effort during the review process.