ReGen: Generative Robot Simulation via Inverse Design

Could you elaborate on the realism of the generated scenarios? If the generated scenarios are not reasonable, off-the-shelf VLM understanding may perform poorly, as observed in Table 3.

• We agree that realism plays a crucial role in ensuring the generated scenarios align with human expectations and enable better performance for off-the-shelf VLMs. However, we observed that VLMs frequently responded with deceleration as the default action. This behavior, consistent with prior findings [A1], indicates VLMs struggle to reason through nuanced spatial and situational situations. In contrast, scenarios generated by DriveCoT (which also uses the CARLA simulator) involve spawning objects directly in the ego-vehicle’s path, requiring a deceleration response. Here, the VLM's default bias aligns with the expected behavior. The observed failures highlight a limitation of off-the-shelf VLMs in reasoning about complex driving situations, rather than pointing to an inherent lack of realism in the scenario design. Importantly, this limitation cannot be attributed to insufficient input, as the VLM is provided with privileged information, including the location and speed of other cars.

• We conducted a preliminary human study to evaluate the realism of the generated simulations. In this study, participants assess whether the scenarios are both realistic and consistent with their descriptions. We will include the analysis of the results from the study in a future revision.

A1. S. Sreeram, T.-H. Wang, A. Maalouf, G. Rosman, S. Karaman, and D. Rus, “Probing Multimodal LLMs as World Models for Driving,” arXiv.org, 2024. https://arxiv.org/abs/2405.05956.

We thank reviewer hjAh for their thoughtful feedback and for highlighting areas that could benefit from further explanation. Below, we provide additional experimental studies and discussions to address these concerns.

The causality reasoning and diversity aspects appear to be inherent properties of the LLMs rather than unique contributions of the framework

• Diversity: Changing models, adjusting hyperparameters, or relying on prompt-engineering methods like ChatScene, which uses GPT-4, result in only marginal improvements in scenario diversity. This is because these methods are limited by the context specified in their prompts, restricting the scope of generated scenarios. In contrast, our method expands the range of possible scenarios by proposing new, potentially unrelated contexts, which are then validated for plausibility using an LLM as a classifier, resulting in significantly greater diversity (Table 1-2).
• Causal reasoning: Our framework uses the inherent causality reasoning capability of LLMs to ensure that generated scenarios are realistic. Here, we define a realistic scenario as one in which cause-and-effect relationships are logical—for example, an "emergency vehicle behind the ego vehicle" causing the "ego vehicle to change lanes."

Generation capabilities seem heavily dependent on the underlying simulator's features

We appreciate the reviewer's observation regarding the dependence on the underlying simulator’s features. However, we view this as a strength of our method: its ability to achieve significantly greater utilization of the simulator’s features compared to prior approaches. For example, as demonstrated in Table 1 and Figure 5, our method achieved greater scenario diversity than ChatScene—which also generates scenarios within the same simulator.

Our results highlight that our method not only adheres to the constraints imposed by the simulator but also leverages its features more effectively to produce a broader range of scenarios. Moreover, as shown in Table 2 and Figure 3, we illustrate that our approach is not confined to a single simulator or domain but is extendable across various simulation platforms.

There is no analysis of the generated graph, only the results are shown, making it unclear whether the main contributions come from the LLM rather than the proposed graph structure.

Figure 7 presents an ablation study showing the impact of changing event-to-event nodes on scenario diversity. Specifically, the pairwise diversity distribution of generated scenarios is categorized by the proportion of scenarios with unique event-to-event edges: 100%, 80%, 40%, and 0%. A value of 100% indicates that all compared scenarios have completely distinct causes, while 0% means they share the same cause but differ in properties such as start location or behavior.

Notably, the bimodal distribution observed in the results highlights two key effects: introducing new event-to-event nodes significantly enhances diversity by expanding the range of causal relationships, while modifications to property-to-entity nodes result in more nuanced variations. Together, these mechanisms offer greater control over the scope and granularity of the generated scenarios. This analysis has been incorporated into the manuscript in the second paragraph of Appendix A.3.

Additionally, there is no ablation study on the graph; for instance, it is unclear what would happen without edge construction.

Thank you for your feedback. We conducted an ablation study to evaluate the accuracy of edge creation during the graph expansion stage, with results presented in Table 4 and further detailed in the first paragraph of Appendix A.3.

• Detail: We assessed event-to-event edges by testing the LLM’s ability to distinguish between causal and non-causal variables. For entity-to-event edges, we evaluated whether the LLM accurately identified simulatable events and selected appropriate assets, consistently demonstrating high performance. Finally, for property-to-entity edges, we tested the LLM’s ability to select the most plausible location, which posed a greater challenge compared to simpler properties, such as determining whether a siren should be on, where it performed more reliably.

In line 161, what description is used when inferring the property node from the source node?

Property nodes specify attributes for each entity, including elements such as location, which is dynamically generated using an LLM (e.g. “in the left lane in front of the ego-vehicle”), or the possible states of an entity in the simulator, which are retrieved from the asset database (see Appendix 1.1–1.2 for additional details). For example, candidate property nodes for a traffic light include all its possible states in the simulator, such as “red,” “green,” “yellow,” and “off.” These proposed nodes serve as candidates for the edge construction step. Examples of this process can be found in Appendix A.1.2. Furthermore, the original manuscript has been updated in the 'Node Proposal' section of the Method to incorporate this clarification.

In line 155, is there an example that clarifies what the prior represents exactly?

The prior refers to external constraints or preferences that guide the node proposal process. Specifically, the prior can take the form of a word or a list of words derived either from (i) human preferences, (ii) or from the entities available within the simulation environment. For example, if "police car" is an entity in the simulation engine, the prior guides the LLM to generate candidate nodes related to this entity. The corresponding prompt might be, “Keywords for police car.” In response, the LLM could propose candidate nodes such as “police chase.” These candidates are then evaluated during the edge construction step to ensure they align with plausible causal relationships to the source node, e.g., “ego vehicle to stop ← police chase.“

This paper demonstrates a plausible generation ability; however, it lacks analysis of the entire generated graph. Could you provide the complete graph generated by ReGen?

We appreciate the reviewer’s feedback regarding the need for more comprehensive documentation of our methodology. In response, we provide the complete graph generated by Regen in “Code Example 6” in Appendix A.2.

More detailed examples are provided in Appendix 1.2 for nodes, covering event, entity, and property nodes, and in Appendix 1.3 for edge proposals, including event-to-event, entity-to-event, and property-to-entity connections. Full output examples can also be found in Appendix 5. For code examples, refer to Appendix A1.1 for the asset database, A2 for the finite state machine, and A2.2 for the full scenario configuration example. Additionally, we have also added comprehensive prompt examples for LLMs in Appendix 5. Specifically, Appendix 5.1 contains prompts for node proposals, and Appendix 5.2 includes prompts for edge creation.

Is there an issue with the brackets in line 182? It seems there are multie understandings here. A more precise notation or explanation is needed.

We acknowledge that our explanation was unclear and have revised this section for clarity.

When expanding the event-to-event nodes, if the LLM generates events that the simulator cannot execute, how does the system handle this?

Our method employs an LLM as a general classifier, inherently performing rejection sampling by evaluating the plausibility of candidate connections in the context of the source node, thereby ensuring that only simulatable events are generated.

• Detail: For event-to-event edges, this approach identifies causal relationships. For entity-to-event and property-to-event edges, it evaluates whether the entities (e.g., an ambulance) and their properties (e.g., a siren) are available in the simulation engine. These entities and properties are defined in the asset database (see Appendix 1.1). We have revised the “Event Construction” section of the method to enhance clarity and provide a more detailed explanation. Additional examples are included in Appendix 1.3, and the accuracy of this classifier is presented in Table 4.

We thank reviewer x8VP for their thoughtful feedback and for highlighting areas that could benefit from further explanation. Below, we provide additional experimental studies and discussions to address these concerns.

Do not see a clear strength in terms of novelty

We appreciate the reviewer’s feedback and recognize the need to better articulate the novelty of our approach.

• Previous methods [A1, A2] aim to increase the diversity of simulations by generating new reward functions, which in turn produce more varied behaviors. This process is computationally intensive and inherently constrained by the need to define distinct reward functions for every simulated scenario.
• Our method proposes reusing and adapting existing behaviors (e.g., a trajectory or reward function) to generate novel simulations. Our method achieves this by altering the environmental context in which these behaviors occur. This approach overcomes the bottleneck of prior methods and enables the creation of richer and more varied scenarios, as extensively demonstrated through experiments. The key insight is that while behaviors are inherently limited, the environments in which they occur are far more diverse. For example, the abrupt stopping of a self-driving car can apply to various contexts, such as encountering a red traffic light, responding to a pedestrian stepping into the road, or yielding to an approaching police car with its siren on.

No comparison with human-designed scenarios

One of the methods we compared against is DriveCoT, which employs a rule-based expert policy to generate ground truth labels for driving scenarios selected from the CARLA Leaderboard 2.0, based on the NHTSA crash typology. We consider these scenarios to be designed by human experts. First, we evaluated the diversity of scenarios generated by our method compared to those from DriveCoT, with our method achieving a significantly higher diversity score (Table 1). Second, we assessed the performance of VLMs on scenarios from both DriveCoT and our method (Table 3).

Unclear metrics for scenario complexity beyond embedding diversity

In our study, we measure complexity by assessing how challenging it is for VLMs to respond accurately to a scenario (Table 3). By evaluating VLM performance in these scenarios, we aim to provide a practical metric for assessing scenario complexity beyond embedding diversity. For example, we observed that in lane-change scenarios such as "avoiding debris," "overtaking a slow vehicle," "merging into an open lane," or "swerving to avoid a wrong-way driver," off-the-shelf VLMs often defaulted to deceleration as the primary action. This behavior, consistent with prior findings [A3], indicates a struggle to reason through nuanced spatial and situational situations. In contrast, scenarios generated by methods like DriveCoT (which also uses the CARLA simulator) involve spawning objects directly in the ego-vehicle’s path, prompting straightforward deceleration responses. Here, the VLM's default bias aligns with expected behaviors, demonstrating a lower level of complexity.

The paper inadequately addresses fundamental simulation constraints, as many proposed scenarios can't be fully simulated due to engine limitations, but there is no systematic approach for dealing with these limitations or assessing their impact on practical utility.

In our approach, a scenario is represented as a graph, with nodes representing events, entities, and properties, and edges capturing transitions between them. To ensure plausibility, we employ an LLM as a classifier to evaluate the validity of each edge. We have revised the Methodology section to improve clarity. To address simulation constraints explicitly, we construct an asset database represented as a directed graph, which encodes the capabilities of the simulation engine (see Appendix 1.1).

A1. Zhang, Jiawei, et al. “ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles.” ArXiv.org, 2024, arxiv.org/abs/2405.14062.

A2. Wang, Yufei, et al. “RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation.” ArXiv.org, 2 Nov. 2023, arxiv.org/abs/2311.01455.

A3. S. Sreeram, T.-H. Wang, A. Maalouf, G. Rosman, S. Karaman, and D. Rus, “Probing Multimodal LLMs as World Models for Driving,” arXiv.org, 2024. https://arxiv.org/abs/2405.05956.

Could you provide more details about the specific prompting strategies used for graph expansion?

Thank you for your feedback. We have added comprehensive prompt examples for LLMs in Appendix 5. Specifically, Appendix 5.1 contains prompts for node proposals, and Appendix 5.2 includes prompts for edge creation. Additionally, detailed examples are provided in Appendix 1.2 for nodes, covering event, entity, and property nodes, and in Appendix 1.3 for edge proposals, including event-to-event, entity-to-event, and property-to-entity connections. Full output examples can also be found in Appendix 5. For code examples, refer to Appendix A1.1 for the asset database, A2 for the finite state machine and the full scenario configuration example.

How do you ensure consistency and reliability in the LLM output, and how do you handle cases where the LLM generates invalid or inconsistent relationships?

We set both the temperature and top-p to 0 to increase the consistency in the LLM output, which empirically yields more deterministic results. Our ablation experiment (Table 4) demonstrated minimal variations in its responses.

Could you provide a detailed analysis of the failure modes

Thank you for raising this point. We have updated the manuscript to include a detailed analysis of the failure modes observed in two key stages of our method: (1) the graph expansion stage (Appendix A.3, paragraph 1) and (2) the grounding into simulation (Appendix A.4.2).

About generated task completeness. How do you guarantee that the generated tasks are feasible for a robot or autonomous agent? i.e., that there is a practical solution to the generated tasks

Our approach ensures task feasibility by converting each scenario generated from the graph expansion process into a finite state machine (FSM). The FSM defines a satisfiability problem, which we solve using tools like Google’s CP-SAT solver. The solver finds solutions for the variables such as the x, y coordinates of the start and end positions $(x_0, y_0, x_T , y_T)$, as well as the speed, such that it satisfies the constraints imposed by the FSM. There is a practical solution if the simulation terminates in a state that satisfies the terminal condition of the FSM. The feasibility rate for driving is 80% and manipulation is 78%.

• Example: The FSM shown below is for the scenario “Ego-vehicle stopping ← Ambulance approaching from behind.” The FSM translates low-level trajectory into abstract states for state tracking. For example, the abstract state “Ambulance Approaching” is defined as a constraint of where the ambulance is behind the ego-vehicle and also in motion if behind_ego(“ambulance”) and is_currently_moving(“ambulance”). Additional details on this process can be found in Appendix A.2

fsm = [[(’ambulance1’, ’Ambulance Approaching’), (’ego-vehicle’, ’Ego Driving Steady’)],
 [(’ambulance1’, ’Ambulance Close to Ego’)], 
[(’ego-vehicle’, ’Ego Braking’)], 
[(’ego-vehicle’, ’Ego Stopped Abruptly’)], 
[(’ambulance1’, ’Ambulance Passing Ego’)]] 

Dear Reviewer x8VP,

Thank you for your feedback, we appreciate you getting back to us.

1. How can LLMs effectively validate robot-environment interactions without a in-depth understanding of robot embodiment? Specifically, how can they verify whether objects are reachable within the robot's configuration space?

We would like to clarify that, in our approach, each simulated environment includes a policy for the ego agent to solve and this takes into account task completeness, physical feasability, and object reachability. We use an LLM to propose high-level constraints -- for example, in the scenario "yielding to an ambulance," the LLM propose a constraint that the ambulance should initially be behind the ego-vehicle before passing ahead -- we do not rely on LLMs to verify whether objects are reachable within the robot's configuration space. Our success rate for task completeness for driving is 80% and manipulation is 78%. We have also provided a detailed analysis of failure modes in Appendix A.4.2.

2. What mechanisms ensure that the generated environments are complete and allow for successful task execution by the robot? These concerns point to a broader issue: the lack of a comprehensive validation protocal for the generated environments.

Our setup is slightly different in that we adopt an inverse design approach: given an ego agent behavior, we generate simulated environments where such behaviors could plausibly occur. A key bottleneck in existing methods lies in obtaining effective policies. Our work demonstrates how reusing existing behaviors can enhance the diversity and realism of simulated environments.

Without validation across multiple dimensions (physical feasibility, task completeness, etc.), it becomes impossible to differentiate between failures caused by the robot model versus environmental constraints. Your current implementation, as shown in Figure 3, presents relatively simple manipulation environments that fall short when compared to the complexity and realism offered by existing benchmarks such as ManiSkill, Habitat, and AI2-THOR. These environments demonstrate the level of sophistication required for meaningful robot simulation research.

While our Figure 3 depicts simpler manipulation scenarios, we also address complex and realistic environments in other domains. For example, in autonomous driving, our generated environments include challenging multi-agent interactions such as "yielding to an emergency vehicle" and "picking up a passenger on the sidewalk," which prior methods [A1, A2, A3] did not address.

For manipulation, we generate complex tasks that involve multiple subtasks. For instance, "storing detergent out of children's reach" requires a sequence of actions: "opening storage furniture," "placing the detergent inside," and "closing the furniture door." Additional examples are available on our website (https://sites.google.com/view/regen-simulation).

Furthermore, our method complements existing methods like ManiSkill, Habitat, and AI2-THOR by enabling the reuse of learned policies in diverse simulated environments. This reusability enhances the diversity and applicability of these policies, allowing for the generation of a broader range of simulated environment (Table 1-2, Figure 5).

Thank you again for your time, and we hope you are feeling better and in good health.

Best,

Authors

A1. Zhang, Jiawei, et al. “ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles.” ArXiv.org, 2024, arxiv.org/abs/2405.14062.

A2. Wang, Yufei, et al. “RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation.” ArXiv.org, 2 Nov. 2023, arxiv.org/abs/2311.01455.

A3. S. Sreeram, T.-H. Wang, A. Maalouf, G. Rosman, S. Karaman, and D. Rus, “Probing Multimodal LLMs as World Models for Driving,” arXiv.org, 2024. https://arxiv.org/abs/2405.05956.

In Q1 ... "each simulated environment includes a policy for the ego agent to solve and this takes into account task completeness, physical feasability, and object reachability." ...

How did you obtain the policy? Replicate the agent behavior?

In Q2 ... "Our setup is slightly different in that we adopt an inverse design approach" ...

Your answer does not address this problem. The thing is that the problem raise from such inverse design, so how the inverse design could resolve the problem.

... "our method complements existing methods like ManiSkill, Habitat, and AI2-THOR by enabling the reuse of learned policies in diverse simulated environments." ...

Table 1-2 does not demonstrate the proposed method could complements existing benchmarks. More expriment should be conducted to show evidances that the proposed method how handle complex scenes (e.g., a layout with several rooms, not an "table-top" scene). The complexity in domestic scene is quite different from complexity in automonous driving scenes, the latter is not senstive to arrangements of objects while you altering the environment.

We thank reviewer HrM2 for their thoughtful feedback and for highlighting areas that could benefit from further explanation. Below, we provide additional experimental studies and discussions to address these concerns.

Additional method details

We have clarified that all LLM queries in our method utilize the GPT-4o-2024-08-06 model with a temperature and top-p value of 0. Additionally, we have revised the Methodology section to provide this information at the end of the first paragraph in subsection 2.2. More detailed examples are provided in Appendix 1.2 for nodes, covering event, entity, and property nodes, and in Appendix 1.3 for edge proposals, including event-to-event, entity-to-event, and property-to-entity connections. Full output examples can also be found in Appendix 5. For code examples, refer to Appendix A1.1 for the asset database, A2 for the finite state machine and full scenario configuration example.

Prompt examples

Thank you for your feedback. We have added comprehensive prompt examples for LLMs in Appendix 5. Specifically, Appendix 5.1 contains prompts for node proposals, and Appendix 5.2 includes prompts for edge creation.

Algorithm 1 will very likely result in infinite loop, as there will likely always be at least 1 node with input degree < 1 (unless there are circular cause and effect dependencies).

The potential for an infinite loop arises because the main graph expansion continuously adds new cause-and-effect relationships. However, this process is controlled by user-defined stopping conditions (graph depth or maximum number of nodes). We have clarified this in the manuscript to improve clarity.

L320 - ReGen is able to generate simulated environments by augmenting visual observations like GPS measurement jamming. However there is no detailed description of how the visual observations are augmented. It seems that some tool use ability of LLM is invoked.

In the case of GNSS jamming, the entity is the GNSS sensor, and the property is noise. We leverage an LLM to generate a function that invokes add_gnss_noise from the CARLA API, simulating a "GPS Jamming" scenario. The FSM tracks state transitions, enabling us to identify the appropriate moments during the simulation to call this function. This entire process is fully automated. We have revised the original manuscript to clarify this process and and included examples in Appendix 2 for further illustration.

Why ReGen outperform the baselines

Our method surpasses ChatScene by significantly diversifying the underlying causes for challenging scenarios. ChatScene primarily generates scenarios involving objects crossing in front of the ego-vehicle (e.g., a pedestrian crossing or a car merging), while our method introduces more complex variations, such as a group of pedestrians or a vehicle traveling in the opposite direction. These scenarios empirically were more challenging for the driving policies, such as requiring larger steering adjustments to avoid groups of pedestrians. We have updated the original manuscript to elaborate on this.

Changing models, adjusting hyperparameters, or relying on prompt-engineering methods like ChatScene, which uses GPT-4, result in only marginal improvements in scenario diversity. This is because these methods are limited by the context specified in their prompts, restricting the scope of generated scenarios. In contrast, our method expands the range of possible scenarios by proposing new, potentially unrelated contexts, which are then validated for plausibility using an LLM as a classifier, resulting in significantly greater diversity (Table 1-2).

• Detail: Our approach decouples node proposals from edge construction. A key perspective is that graph expansion systematically explores the LLM's knowledge space, uncovering and organizing latent information into a structured graph. The node proposal component in our method generates new context condition on priors. For example, the prior guides the LLM to generate candidate nodes related to an entity such as a “police car”. The corresponding prompt might be, “Keywords for police car.” In response, the LLM could propose candidate nodes such as “police chase.” Then, during edge construction, the LLM – which acts as a general classifier – evaluates which nodes are plausible within the context to ensure that the resulting scenarios are not only diverse but also simulatable. Together, the node proposal and edge construction steps enable us to explore all potential events and uncover corner-case scenarios.

We thank reviewer q9yg for their feedback and for highlighting areas that could benefit from further explanation. Below, we provide additional experimental studies and discussions to address these concerns.

Primarily focused on diversity and complexity, with less emphasis on the realism and physical accuracy of the generated simulations

Thank you for your feedback. We agree that realism plays a crucial role in ensuring the generated scenarios align with human expectations. We address realism through two aspects.

• First, we ensure the scenario’s realism by validating the plausibility of cause-and-effect relationships between events. For example, a scenario is realistic only if the cause-and-effect is logical, such as an event "emergency vehicle behind the ego-vehicle" causing the event "ego-vehicle to change lanes.”

• Second, we conducted a preliminary human study to evaluate the physical realism of the generated simulations. In this study, participants assess whether the scenarios are both realistic and consistent with their descriptions. We will include the analysis of the results from the study in a future revision.

LLMs can be computationally expensive and may not always generate semantically accurate or physically plausible scenarios Thank you. We address this concern in two parts:

• Empirically, our method takes approximately 1-3 minutes to generate a scenario using GPT-4o-2024-08-06.
  • Details: We classify multiple candidates in a single prompt, reducing the query complexity in each iteration of the graph expansion step to $O(1)$. This approach avoids the $O(n^2)$ complexity of pairwise comparisons, where $n$ is the number of candidate nodes.
• Second, we employ an LLM as a general classifier to evaluate the plausibility of proposed events, entities, and properties within the graph. This method ensures that each graph component adheres to the constraints of the simulation engine.
  • Details: Our edge construction leverages an LLM as a general classifier to evaluate the plausibility of candidate connections in the context of the source node. For event-to-event edges, this process identifies causal relationships. For entity-to-event and property-to-event edges, it verifies whether the entities (e.g., an ambulance) and their properties (e.g., a siren) are supported by the simulation engine. These entities and properties are defined in the asset database (see Appendix 1.1), ensuring compatibility with the simulation of the specified event. We conducted an ablation study (more details in Appendix A.3) to evaluate the accuracy of each edge type.

Other domains not extensively explored

We appreciate the reviewer’s comment regarding the exploration of other domains. Most existing works tend to focus either on driving scenarios [A1, A2, A3] or manipulation tasks [A4, A5, A6], often neglecting cross-domain exploration. In our work, we chose driving and manipulation as they represent two important fields of interest to the robotic and simulation communities. We would welcome the reviewer’s suggestions on additional domains that could benefit from our approach or serve as compelling directions for future work.

Citations:

A1. X. Yang et al., “DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving,” arXiv.org, 2024. https://arxiv.org/abs/2408.00415.

A2. Zhang, Jiawei, et al. “ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles.” ArXiv.org, 2024, arxiv.org/abs/2405.14062.

A3. Hu, Anthony, et al. “GAIA-1: A Generative World Model for Autonomous Driving.” ArXiv.org, 2023, arxiv.org/abs/2309.17080.

A4. Wang, Yufei, et al. “RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation.” ArXiv.org, 2 Nov. 2023, arxiv.org/abs/2311.01455.

A5. Wang, Lirui, et al. “GenSim: Generating Robotic Simulation Tasks via Large Language Models.” ArXiv.org, 2023, arxiv.org/abs/2310.01361.

A6. Hua, Pu, et al. “GenSim2: Scaling Robot Data Generation with Multi-Modal and Reasoning LLMs.” ArXiv.org, 2024, arxiv.org/abs/2410.03645.

Dear Reviewer q9yg,

Thank you very much for your response and for increasing your rating. We appreciate your suggestion regarding human-robot interaction scenarios and would like to highlight some examples from our work that align with this direction. In the driving domain, we have simulated scenarios such as "yielding to an emergency vehicle" and "picking up a passenger on the sidewalk," which involve human-robot interactions that prior methods [A1, A2, A3] were unable to address. Similarly, in the manipulation domain, we have simulated scenarios like "retrieving food from a fridge for a guest," "storing detergent out of children's reach," and "letting a guest out the door." We look forward to further advancing these directions in future work.

Best,

Authors

A1. T. Wang, E. Xie, R. Chu, Z. Li, and P. Luo, “DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving,” arXiv.org, 2024. https://arxiv.org/abs/2403.16996

A2. C. Sima et al., “DriveLM: Driving with Graph Visual Question Answering,” arXiv.org, Dec. 21, 2023. https://arxiv.org/abs/2312.14150

A3. Zhang, Jiawei, et al. “ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles.” ArXiv.org, 2024, arxiv.org/abs/2405.14062.

Thank the authors for replying to my questions. All my concerns are clear now clear. As for additional domains, I suggest the authors to add more human-robot interaction scenarios. I increased my rate to 6.