We appreciate the reviewer for recognizing our contributions of “a new embodied AI simulation platform for large-scale, photorealistic, dynamic urban environments”, “very unique and meaningful” benchmarks. We address your questions and concerns below.

Q1&W1&L1: Sim-to-real

Yes, we fully agree that enabling sim-to-real transfer is crucial for embodied AI research. Although our current experiments focus on simulation, RoboScape was designed with sim-to-real considerations in mind. We plan to inject realistic sensory noise, such as image blurring, depth corruption, and camera jitter.

Q2&W2&L2: Continuous robot action space

Yes, we will. While the current benchmarks use a discrete action space for simplicity and clarity in the two case studies, RoboScape supports a continuous action space through velocity or acceleration commands. Additionally, RoboScape is built on Unreal Engine, which supports integration with ROS2, allowing users to implement low-level dynamics and control if needed.

Q3&W4&L4: Selection of baselines

We did conduct preliminary experiments with standard RL-based baselines (e.g., PPO) and a hybrid baseline. However, these approaches performed poorly in our proposed case studies, likely due to the complexity of instruction-driven tasks and the need for long-horizon reasoning. Our focus is on developing a flexible and extensible simulation platform. While important, designing high-performing policies for these tasks would require significantly more sophisticated architectures and task-specific tuning beyond the current state-of-the-art, which is out of the scope and orthogonal to the primary contribution of our paper.

Having said that, we have conducted additional experiments during rebuttal to include several non-VLM baselines and will include them in the revised Table 2:

• Hybrid Baseline: We have evaluated the performance of a hybrid planner where a VLM serves as the high‑level decision maker—deciding whether to continue straight down a street until the next intersection or to turn at that intersection—while A* is used as a low‑level path planner to generate and execute the movement commands. The subtask success rate is 32.53%.
• RL Baseline: We utilize two NVIDIA L40S GPUs, each running two parallel instances of RoboScape. Following the setting of VLN-CE [1], the agent is powered by a multimodal policy that combines DeBERTa-v3 for instruction encoding and DINOv2 for visual perception. We first pretrain the policy through imitation learning (specifically, behavioral cloning) on expert demonstrations, and then further finetune it using PPO. Rollouts are collected concurrently across all environments. The current RL baseline achieves a 28.37% subtask success rate, highlighting the challenging nature of our tasks. We believe RoboScape provides a strong foundation for future research on more advanced RL methods, by offering parallelizable, photorealistic, large-scale, and dynamic city environments, along with the ability to collect and incorporate human demonstrations that naturally include errors and corrective behaviors.

Q4: Compute requirements

To ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops.

Recommended Setup

• CPU: Intel Core i7‑12700H or AMD Ryzen 9 5900HS
• GPU: NVIDIA RTX 3070 or GPU with more than 6 GB
• RAM: 32 GB

Minimum Setup (60 FPS for RoboScape)

• CPU: Intel Core i7‑11300H or AMD Ryzen 9 4800H
• GPU: NVIDIA RTX 2060 (notebook)
• RAM: 16 GB

Q5: Release the full simulator and benchmark code

Yes, we plan to release the full simulator and benchmark code. Upon publication, we will open-source all binaries, the Gym wrapper, and provide both a Windows executable and a Singularity container for Ubuntu and macOS, allowing users to run RoboScape out of the box without compilation.

W3&L3: RoboScape-20K Dataset

Indeed, current training data only consists of expert (oracle) trajectories. However, we emphasize that RoboScape is fully compatible with reinforcement learning, which could potentially address the limit as shown in the additional experiments. Furthermore, RoboScape provides a human interface for collecting trajectories from human users, which naturally include suboptimal and error-correcting behaviors. This enables future work to explore learning from human corrections or noisy demonstrations.

W5&L5: Rule-based traffic simulation

All pedestrians and vehicles in RoboScape are independently controllable and support asynchronous control. By applying more sophisticated control policies to each pedestrian, RoboScape is capable of supporting highly realistic traffic simulations. Currently, the traffic system is tailored to support our two case studies, primarily to demonstrate RoboScape’s capabilities as a flexible and extensible platform.

W6&L6: Scalability

While the average episode length in our current experiments is around 500 meters, this setting already reveals significant performance degradation across baselines, highlighting the challenge even at this scale. Importantly, RoboScape is built with scalability in mind: it supports procedural generation of much larger urban environments and can accommodate significantly longer episodes. This design ensures that researchers can easily stress-test navigation agents in more complex, long-horizon tasks, which we plan to explore in future benchmarks.

References:

[1] Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Thank the authors for thoughtful rebuttal and conducting new experiments. My concerns are resolved and looking forward to seeing the opensource!

We appreciate the reviewer for recognizing our contributions of “a novel simulator that addresses the lack of realism, scalability, and versatility in existing urban simulators” and “two new benchmarks provide a comprehensive evaluation of robot capabilities in realistic scenarios.” We address your questions and concerns below.

Q1: Integration of New AI Models

Currently, RoboScape uses pre-defined 3D assets, primarily obtained from Unreal Engine Marketplace (FAB), which we further customize for integration. However, the framework allows easy integration of generative methods such as DreamFusion to synthesize 3D assets from text, and SMPL-X-based text-to-body motion generation models to generate human motion sequences from natural language descriptions. We are actively developing these extensions to further enhance RoboScape’s ability to support diverse AI research.

Q2: Specific Computational Requirements

To ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops.

Recommended Setup

• CPU: Intel Core i7‑12700H or AMD Ryzen 9 5900HS
• GPU: NVIDIA RTX 3070 or GPU with more than 6 GB
• RAM: 32 GB

Minimum Setup (60 FPS for RoboScape)

• CPU: Intel Core i7‑11300H or AMD Ryzen 9 4800H
• GPU: NVIDIA RTX 2060 (notebook)
• RAM: 16 GB

W1: Limited Action Space for Human Agent

We agree that the action space can be further expanded. By defining poses, it is convenient to add more actions within our framework. As an initial effort, we have added 5 new types of human actions commonly observed in daily life: (1) picking up, carrying, and dropping objects; (2) sitting down and standing up; (3) entering and exiting vehicles; (4) opening and closing containers (e.g., boxes, cabinets, car trunks); and (5) engaging in social interactions (e.g., chatting, discussing, arguing).

We achieved this by using pre-defined human body motion for each type and adjusting the poses using inverse kinematics for human-object interactions (e.g., adjusting hand poses to grasp objects). We have also implemented a flexible API to enable these actions to be triggered not only by rule-based planners but also by other models such as LLMs, enabling more diverse and context-aware human behaviors. In ongoing work, we are exploring the use of generative models such as text-to-body motion for producing realistic human actions beyond the predefined space. This will allow RoboScape to support richer and more human-like behavior generation in the future.

We are happy that the reviewer considered our work “significant to the field,” and we address your questions and concerns below.

Q1: RoboScape Details

Thanks for the suggestion. We further clarify the details of RoboScape below and will include them in the revised Section 3 and Appendix A.

• Underlying simulation engine. Our simulation is built on Unreal Engine 5, leveraging its native Chaos physics pipeline: each object is assigned an appropriate collision mesh, and at each fixed simulation tick performs discrete‑time integration of Newton’s equations—resolving forces, collisions, and joint constraints via its iterative solver.
• Python API and runtime environment. On top of UE5, we’ve implemented a dedicated Python layer that communicates with the engine through an updated UnrealCV‑based TCP server, where high‑level commands issued in Python are forwarded over TCP straight into the UE5 runtime. On top of this API, we provide a standard Gym interface so that researchers can plug in and benchmark any baseline with minimal effort. We will distribute a Windows executable and a Singularity container for Ubuntu and macOS so that users can run RoboScape out of the box without local compilation; all binaries, container images, and the Gym wrapper will be open‑sourced upon publication.
• Hardware requirements and simulation/rendering performance. To ensure wide accessibility, RoboScape supports adjustable rendering resolutions, allowing deployment across both high-end servers and modest laptops.

Recommended Setup

• CPU: Intel Core i7‑12700H or AMD Ryzen 9 5900HS
• GPU: NVIDIA RTX 3070 or GPU with more than 6 GB
• RAM: 32 GB

Minimum Setup (60 FPS for RoboScape)

• CPU: Intel Core i7‑11300H or AMD Ryzen 9 4800H
• GPU: NVIDIA RTX 2060 (notebook)
• RAM: 16 GB

We evaluated all baselines on a headless machine with an AMD EPYC 9534 CPU, L40S GPU, 64 GB RAM, and 720×600 resolution. We can run 2 instances in parallel with a fixed 60 fps. Here’s the table when we stress-test the runtime performance with different settings.

• Human‑in‑the‑loop interface. We support a human‑in‑the‑loop interface through which a human operator uses a mouse and keyboard to control the robot; all trajectories are recorded automatically as expert demonstrations for downstream training.

Q2: Baselines Evaluation under Different Settings

To clarify, our ROBOSCAPE‑MMNAV benchmark includes two distinct task settings to isolate how different conditions affect baseline performance in Table 2 and Table 3 at line 196. In the easy setting, all static obstacles (trees, benches, fire hydrants) and dynamic agents (traffic, pedestrians) are removed, so the challenge reduces to navigation in static environments. In the hard setting, we introduce both static obstacles and dynamic obstacles under sunny conditions. While baseline models achieve similar overall success rates and progress scores in both settings, they are unable to effectively avoid static and dynamic obstacles, consistently fail to yield to pedestrians, and do not adhere to basic traffic rules. None of the baselines could reliably complete the task under sunny setting and we could further anticipate that under more adverse weather or lighting condition—overcast skies, rain, or added visual occlusions—their performance would degrade even more.

W3: Lack of New Methods & Ablation Experiment

The goal of this work is the development of a highly customizable, photorealistic, city-scale simulation platform designed to provide the embodied AI community with training and testing environments that closely mirror real-world scenarios (Line 44). We further introduce two new benchmarks as case studies to illustrate RoboScape’s functionalities and versatility. Developing new methods is out of the scope of this work. We believe that our platform can be a valuable resource for the community to develop new methods in future work.

We conducted new ablation experiments to assess the contribution of each input modality, using the same GPT‑4o backbone. Our basic setting includes a ground truth segmentation input, explicit ReAct prompt framework [2] and separate perception-action calls which simplify the mapping. While none of the implementations successfully reach the destination, we observe that removing the GT segmentation image, ablating the ReAct framework, or merging the perception-action calls into a single inference would decrease the performance of the baseline. Furthermore, the depth image and stripped vision input hurdle the performance.

We will include the following results in the revised Section 4.2 using an additional table.

Performance summary (subtask success rate):

• GPT‑4o: 34.38%
• Merged perception-action call:33.54%
• w/ depth: 33.39%
• w/o explicit ReAct framework: 32.21%
• w/o GT segmentation: 31.90%
• w/ stripping: 31.22%

Analysis:

• The perception-action framework simplifies the mapping process and reduces hallucinations.
• The model already has a certain depth prediction capacity. Adding depth images does not help significantly and may even introduce noise if the colormap is not well aligned.
• ReAct framework does help reduce hallucination and stabilize the policy in complex scenarios
• GPT-4o lacks sufficient training on intersection-related scenarios, but the GT segmentation image helps mitigate this limitation.
• Although shown to be promising in [2], stripping the observation into vertical chunks increases the difficulty of image matching, which is crucial to our current simple settings.

References

[1] ReAct: Synergizing Reasoning and Acting in Language Models
[2] NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models

We are happy that the reviewer considered our work “has better photorealism than existing urban simulators” and the two case studies “are practical in the real world while less studied in the urban embodied settings”. We address your questions and concerns below.

Q1: Running Efficiency of RoboScape

RoboScape supports running multiple instances in parallel in headless mode for efficient RL training.

We evaluated all baselines on a headless machine with an AMD EPYC 9534 CPU, L40S GPU, 64 GB RAM, and 720×600 resolution. We can run 2 instances in parallel with a fixed 60 fps. Here’s the table when we stress-test the runtime performance with different settings.

Q2 & W1: Selection of Baselines

We appreciate the reviewer’s suggestion regarding the inclusion of task-specific models such as robot navigation systems or RL-based policies. We have conducted preliminary experiments with standard RL-based baselines (PPO). However, these approaches performed poorly in our proposed case studies, likely due to the complexity of instruction-driven tasks and the need for long-horizon reasoning. Designing high-performing policies for these tasks would require significantly more sophisticated architectures and task-specific tuning—an important direction for future work, but one that is orthogonal to the primary contribution of our paper. Our primary contribution lies in the introduction of RoboScape, an open-source and extensible simulation platform that supports customizable, diverse scenarios for evaluating embodied agents.

We have conducted the following additional experiments and will include them in the revised Table 2:

• Hybrid Baseline: We have also evaluated the performance of a hybrid planner where a VLM, GPT-4o, serves as the high‑level decision maker—deciding whether to continue straight down a street until the next intersection or to turn at that intersection—while A* is used as a low‑level path planner to generate and execute the movement commands. The subtask success rate is 32.53%.
• RL Baseline: We utilize two NVIDIA L40S GPUs, each running two parallel instances of RoboScape. Following the setting of VLN-CE [1], the agent is powered by a multimodal policy that combines DeBERTa-v3 for instruction encoding and DINOv2 for visual perception. We first pretrain the policy through imitation learning (specifically, behavioral cloning) on expert demonstrations, and then further finetune it using PPO. Rollouts are collected concurrently across all environments. The current RL baseline achieves a 28.37% subtask success rate, highlighting the challenging nature of our tasks. We believe RoboScape provides a strong foundation for future research on more advanced RL methods, by offering parallelizable, photorealistic, large-scale, and dynamic city environments, along with the ability to collect and incorporate human demonstrations that naturally include errors and corrective behaviors.

Q3: Customization

High customizability is a core feature of RoboScape. We will outline how to create new tasks, scenarios, and embodied agents in Appendix A.4 to support flexible benchmark extension. The two benchmark tasks included in the paper are intended as case studies to demonstrate the functionalities and utilities of RoboScape. RoboScape was explicitly designed to support user-friendly customization of both tasks and environments.

• New scenarios. Users can generate diverse and realistic urban layouts through our Python API by providing simple metadata inputs (e.g., number of streets, street length, object categories, and their spatial distributions, such as “10% trees, 5% tables and chairs”). This allows researchers to create arbitrarily large and varied city environments.
• New tasks. Defining tasks in RoboScape is similar to standard Gym environments, as the Python API of RoboScape follows the same format for agent control (including pedestrians and vehicles). Users can spawn different types of agents, customize observation spaces (e.g., RGB, depth, or semantic segmentation) and action spaces (continuous or discrete actions), and program new goal definitions (e.g., language instructions, target images, or spatial objectives). Additionally, while the current pedestrians follow rule-based logic, users can override behaviors to simulate complex or rare cases. For instance, one can simulate a jaywalking pedestrian by scripting a few agents to cross during a red light, while placing a robot at the intersection to evaluate its reaction.

W2: New Embodied Robot

We fully agree that many other types of embodied agents (e.g., delivery robots, wheelchairs, humanoids, or drones) are relevant in urban settings. In RoboScape, we have included an additional embodied agent—a scooter—that can be controlled using velocity and angular velocity commands through our Python API, and it will be included in the revised Section 3.2.

To add a new robot type in RoboScape, one can leverage the existing Python API of RoboScape to conveniently customize action spaces—either continuous or discrete—as well as observation spaces such as RGB, depth, or semantic segmentation images. The required additional work (1) obtaining a new robot asset, typically from the Unreal Engine Marketplace; (2) defining the robot's actions using Unreal's Blueprint system; (3) integrating these actions with our Python API to enable high-level control; and (4) attaching our camera components to support the desired observation space.

W3: Action Space in RoboScape

Current action space in RoboScape is consistent with other urban simulators like CARLA and MetaUrban, which adopt a similar high-level action space. Additionally, RoboScape is built on Unreal Engine, which supports integration with ROS2, allowing users to implement low-level dynamics and control if needed. This makes it feasible to reduce the sim2real gap by customizing robot-specific control pipelines.

W4: Details of Dataset RoboScape-20K

Thanks for the suggestion. We will move the details of RoboScape-20K from Appendix D.4 to the main paper. To further clarify, RoboScape-20K is a training set consisting of 20,000 robot navigation cases for RoboScape-MMNav. The training environments are entirely disjoint from the evaluation environments, and 33% of the building assets used during evaluation are intentionally excluded from the training set to ensure a clean generalization split.

For each training case, the input is the robot’s current observation, and the label is the correct next action to take. Since our refined pipeline includes a reasoning module, we additionally provide optional intermediate supervision signals—such as the correct facing direction and the ground-truth distance to the goal—to encourage chain-of-thought learning. These auxiliary labels are optional and can be ignored depending on the training paradigm and model architecture.

Paper Formatting

We will merge the appendix and supplementary materials into a single, complete appendix in the final version.

References:

[1] Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

We appreciate your thoughtful suggestion for deeper analysis. As observed, both hybrid and RL baselines perform poorly even on the easy split. This stems from the task’s demand for embodied spatial reasoning and multimodal instruction grounding.

Additional Analysis for non-VLM-based Approaches

Compared to the initial behavior cloning training (which only had 11.61% subtask SR), RL finetuning can significantly boost the performance to 28.37%. However, we observe that RL policies frequently failed at recognizing the correct intersection and making the right turn. This suggests the challenges of conducting visual and spatial reasoning in realistic environments simulated by RoboScape. The distance to subtask SR correlation for RL is strongly negative (-0.522). This could be caused by sparse rewards and insufficient exploration on longer trajectories.

Compared to GPT-4o baseline, the hybrid baseline performs better on turning (the success rate for turning increases from 42.90% to 66.60%), since once the position is correct, the planner executes turns reliably. However, it performs worse on intersection matching (decreases from 17.50% to 7.70%) because it currently has only one opportunity to match at the corner, with less tolerance than VLM methods that benefit from fine grained action updates.

Why is the performance worse than Multi-Robot Tasks

The multi-robot search task shifts emphasis to scene description and effective communication. The main robot has the oracle planner which covers a substantial portion of the route, and repeated communications allow adjustment. The condition for success, to spot the other robot, is also less strict than the single-robot setting.

More Results on Failure Modes

We analyze failure models for each subtask (we define subtasks in Sec. 4.1). For VLM baselines evaluated in the current benchmark, we further identify and analyze failure modes within each type of subtasks. Below are the most frequent GPT-4o failure modes across these subtasks.

Table 1. Most frequent GPT-4o failure modes

Among these failure modes, 1.1, 2.1, 2.3, 3.2 and 3.3 are all related to embodied reasoning since these scenarios involve locating oneself based on the egocentric view, especially 2.1, where the robot must reason when and how to turn. Failure mode 1.3 and 3.1 are related to spatial reasoning, while 1.2 reflects basic visual grounding errors and 2.2 shows that the performance can further benefit from enhanced planning and memory capacity. Therefore, most failure modes revolve around spatial and embodied reasoning, which are aligned with our original focus on VLM-focused evaluation.

We will expand Sec. 4.2 with more results and analyses regarding failure modes and types of subtasks.

Additional Reasoning Model Results

To further validate our benchmark, we additionally tested o3 and o3 pro. The successful trials show that with correct reasoning and accurate spatial understanding, the model can complete the task end-to-end. All successful trajectories contain at least 60 steps and include all task types, which reduces the chance of overly simple outliers.

Table 2. Additional Benchmark metrics

The results indicate that improved reasoning abilities boost performance. In our experiment the reasoning models show improved depth estimation and destination alignment, which further demonstrates the importance of visual and spatial reasoning in our benchmark.

Thank you for the constructive suggestions. All the additions above will appear in the revision.