Thank you for your insightful feedback. We are glad that you share the excitement of the implication of this to the potential of scaling up the training of embodied generalist agents.

How can this be applied to the field of robotics and embodied AI?

We present a novel method for factorizing simulation codebases, based on factorized POMDP representation, which allows for efficient context selection. Since most robotics tasks can be formulated as a POMDP, we believe that our method can beneficial in the field of robotics and embodied specifically in assembly related tasks at they require many dependent steps. To further demonstrate the superiority of our method, we also conduct additional experiments on 12 assembly-related tasks where FactorSim outperforms all Chain-of-Thought baselines used in GenSim by a large margin due to its ability to factorize the multi-step process into subtasks and generate these subtasks in separate steps, by using only the relevant context needed. Please find more details in the following section.

Details on the robotics experiment

Refer to Figure 2 in the newly attached pdf for an overview of our robotics experiment. Below, we explain our experimental settings in more detail and how we apply FactorSim to generate robotics tasks.

The paper GenSim [1] proposed a benchmark to investigate how well Large Language Models (LLMs) can design and implement simulation tasks and how to improve their performance on task generation. This benchmark includes a list of tasks described in the following format:

• Task: Build-Car
• task-name: build-car
• task-description: Construct a simple car structure using blocks and cylinders. assets-used: [block/block.urdf, ball/ball-template.urdf]

After the task code is generated, GenSim builds on top of CLIPort, evaluates the tasks by feeding them through a series of checkers that validate "syntax-correctness," "runtime-verification," and "task-completion."We followed the same experimental setting as described in their paper, except that we introduced an additional metric, "human-pass-rate" to measure for prompt alignment. This “human pass rate” checks whether the generated task adheres to the input prompt upon completion.

FactorSim achieved state-of-the-art performance on their benchmark. To clarify how we apply FactorSim to the benchmark, we first describe how the POMDP is defined in this setting:

• States: object states in the environment (color, object pose, size, scale)
• Reward function: a set of functions that define how the rewards are given (e..g., a reward of X is given when the blue cube is in the bowl). This is analogous to the scoring functions are generated in the case of RL game generation.
• Transitional function and Observation function: The underlying physics simulation and the robot assumed in CLIPort handle these aspects, so we don't generate them.

We apologize if we didn’t make it clear how we applied FactorSim to generate robotic tasks. To summarize, a robotic task is no different than a RL game in that the final robotic task can be modeled as a POMDP. While the transition dynamics in this case are often handled by a physics simulator, we still need to generate the “reward function.” The only part left to generate is essentially the reward function, in the form of goal states. Objects in the scene and their arrangements are essentially state variables, and the functions are essentially functions that operate on these state variables to define whether a state should be given rewards. This is analogous to how in RL games we implement a reward function that takes in a set of state variables and updates them.

Additional robotics task generation experiment

To further showcase FactorSim's effectiveness in generating robotics tasks, we form a list of 12 assembly tasks (i.e. build related tasks) and by combining the assembly tasks in GenSim's benchmark and by asking GPT-4 to generate additional assembly tasks such as "BuildLampPost" or "BuildDogHouse". As shown in Table 1 of the attached pdf to our general response, FactorSim outperforms all baselines by a large margin, including GenSim's two Chain-of-Thought baselines. We also provide visualizations of tasks that FactorSim is able to generate, demonstrating collection from using the oracle agent in the accompanying figure. These tasks use primitive assets like blocks and balls to form more complicated structures. We believe that scene generation fits into the POMDP framework, similar to what we achieved in our robotics task generation experiment.

Can we improve upon the results in Table 1 and Figure 3

The low success rates testify to the difficulty of the tasks in the benchmark. We additionally ran AgentCoder [1], a SOTA code generation method that uses a multi-agent system to perform CoT prompting, on our benchmark. Please find the results in the General Response. While AgentCoder claims to refine code iteratively, it performs poorly because it relies on a test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark, the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard "role-based" Chain of Thought decompositions.

How is the background generated?

In Pygame learning environments, the game states are represented as a non-visual state, as exemplified by the documentation of the Catcher game: https://pygame-learning-environment.readthedocs.io/en/latest/user/games/catcher.html.

[1] Huang, Dong, et al. "Agentcoder: Multi-agent-based code generation with iterative testing and optimisation." arXiv preprint arXiv:2312.13010 (2023).

Dear Reviewer 6Px5,

Thank you for taking the time to read our rebuttal. We have refined our figures, as you suggested.

Best regards,

Authors.

Thank you for dedicating your time to review our paper and for providing insightful feedback. We are glad to learn that you find our paper to be well-written and comprehensively evaluated.

Novelty

We present a novel method for generating coded simulations that allows for efficient context selection; our method outperforms the Chain-of-Thought baseline in generating complex code, e.g., RL environments by a significant margin (See Table 1 in the main paper) and robotics tasks (See Figure 3 in the main paper and Table 1 in the pdf attached to our general response). More specifically, our method differs from the existing Chain-of-Thought methods by introducing a principled way to select context from the codebase, based on the factorized POMDP representation. It can be applied to any code generation task where the target generation can be modeled as a Partially Observable Markov Decision Process (POMDP), which is a flexible representation. The key advantage of our proposed representation is that it helps us model dependencies between different state variables and functions, instead of having to consider the entire codebase when making modifications to it or having to retrieve context code snippets based on some pre-defined similarity scores (e.g., akin to Retrieval Augmented Generation). To further showcase the novelty and effectiveness of FactorSim, we ran an additional SOTA baseline that uses a "role-based" chain of thought to perform code generation.

While AgentCoder claims to refine code iteratively, it performs poorly because it relies on a test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark, the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard "role-based" Chain of Thought decompositions, and that it is non-trivial to generate simulations from complex textual specifications.

Failure modes and Limitations

The improvement of FactorSim stems from better context provided in the prompts, allowing the LLMs to be more focused during code generation. Below, we discuss one failure mode and one limitation of FactorSim. The main failure mode we observe with FactorSim is when the context is selected incorrectly, leading to incorrect implementation. However, we find that the benefit of FactorSim outweighs the occasional errors LLMs make when selecting contexts, as shown in our experiments.

In our experiment of generating robotics tasks, we find that all baselines, including our method, often ignore physical constraints necessary for the task to be completed by a robot. It is difficult for LLMs to consider context related to these "constraints" necessary for the task to be completed without being explicitly prompted. For example, if the LLM is prompted to generate the task "build a bridge", LLMs might generate a "bridge" block that is too small to span the two bottom items' distance. When the LLM is prompted to generate the task "put the ball in the container," the generated task might consist of a base container that is much smaller than the size of the ball. We leave this to future work.

Thank you for your feedback! We are glad that you find this an important problem and that you find our evaluation solid and comprehensive.

Clarification of our Motivation and Novelty and why we chose Robotics as the primary area

Recent advancements in foundational models have demonstrated their value in cognitive planning in robotics. The use of these models for obtaining lower-level policy is relatively underexplored, with many works focusing on direct control outputs [6, 7]. Our work leverages foundational models to generate simulations for training (i.e., Generative Simulation), a promising approach for training RL and robotics agents [1, 2]. However, LMs often fail at generating large amounts of complex code, leading to incorrect simulations and significant distributional shifts in downstream environments.

By representing our codebase as a factorized POMDP, we can decompose the code into simpler chunks, reducing the need for extensive context. This method improves the accuracy of generating simulations with complex logic, outperforming all Chain-of-Thought-based baselines in generating prompt-aligned coded simulations. In zero-shot transfer experiments, more accurate code generation for simulations proves pivotal for the downstream training of RL and robotics agents. FactorSim’s zero-shot performance reaches 50% of direct training results in 5 out of 8 environments, whereas baselines show significant transfer in only 1-2 environments. Given that most robotics tasks can be formulated as a POMDP, our method could significantly impact the robotics community.

Presentation

Thank you for your detailed feedback on our presentation. We understand the importance of legibility and have made the following improvements to our paper:

• We included intuitive explanations for our method. We propose Factored POMDP representations to model an existing simulation codebase as a hypergraph, with nodes as state variables and hyperedges as functions relating to one or more state variables. Based on instructions for module additions, LLMs first select relevant state variables, then include only these variables and related functions in subsequent prompts.
• We added a new figure to demonstrate how our formulation directly corresponds to prompt construction, with general explanations and concrete examples.
• We included two new figures to explain our robotics task generation experiment and show visualizations of tasks that FactorSim successfully generates, which all other baselines fail to achieve.
• We moved the world model part of the related section to the appendix.

Clarification on our Benchmark

Pygame Learning Environment [3] is a Reinforcement Learning benchmark used in various existing works [4,5]. We extend these environments to include prompts with detailed specifications of game logic paired with system tests to test for “precise” prompt adherence ability in simulation code generation. When coupled with the original RL environments, our extension enables the evaluation of not only the accuracy of the generated code but also its usefulness in solving RL tasks (i.e., zero-shot transfer).

Additional experiments on the benchmark

We additionally ran AgentCoder [8], a SOTA code generation method that uses a multi-agent system to perform CoT prompting, on our benchmark.

Additionally, we added two Atari games to our benchmark to increase the difficulty of our tasks. Here are provide some preliminary results:

This benchmark allows us to investigate whether classical RL environments can be solved through the paradigm of generative simulation.

In figures like figure 6, is the human pass rate based on the previous stage e.g. only executable code.

Yes. These four metrics (i.e. "syntax-correct", "runtime-verified", "task-completed", "human-pass") have an incremental structure from syntax to runtime to successfully generate demonstrations to human verification where failing the former metrics implies failing on the latter metrics.

Explain the insights re: open source models’ performance

In our experiments, llama3 performs very well on generating specific tasks but exhibits instability, performing exceptionally on some tasks while poorly on others. Empirically, we observe that it resorts to "memorization" more, suggesting that if the memorized code is correct, it excels at generating those tasks. GenSim with CoT, both bottom-up and top-down, are competitive baselines that use a chain of 4-7 prompts. Despite this, FactorSim outperforms it, demonstrating its superior performance in task generation.

[1] Xian, Zhou, et al. "Towards generalist robots: A promising paradigm via generative simulation."

[2] Katara, Pushkal, Zhou Xian, and Katerina Fragkiadaki. "Gen2sim: Scaling up robot learning in simulation with generative models."

[3] Tasfi, Norman. "Pygame learning environment." GitHub repository (2016).

[4] Riemer, Matthew, et al. "Learning to learn without forgetting by maximizing transfer and minimizing interference." ICLR 2019.

[5] Tucker, Aaron, Adam Gleave, and Stuart Russell. "Inverse reinforcement learning for video games."

[6] Wang, Yen-Jen, et al. "Prompt a robot to walk with large language models."

[7] Nasiriany, Soroush, et al. "Pivot: Iterative visual prompting elicits actionable knowledge for vlms."

[8] Huang, Dong, et al. "Agentcoder: Multi-agent-based code generation with iterative testing and optimisation."

I acknowledge that I've read the rebuttal and other reviewers' opinions.

Figure 1 in the rebuttal pdf definitely made the paper much more intuitive, allowing me to verify the original interpretation of the paper is correct. I recommend the authors to further improve its quality and put it in the paper whether it's accepted to Neurips or not.

I still believe robotics should not be the primary area for reviewer allocation even given the author's response - just as reviewer EKr6 mentioned, the robot experiments seem like an afterthought. LLM / code synthesis shall be the better pool of reviewers, and I believe the AC shall consider this when making the final decision from our reviews.

I am raising the score for the presentation & overall rating a bit since I find the general response/pdf helpful to understanding, yet still deem the paper unable to meet the acceptance threshold. I am willing to defend my rating, although other reviewers may have different opinions.

Thank you for taking the time to review our paper and for providing constructive feedback. We are glad you share our excitement about FactorSim's performance on the challenging task of zero-shot transfer. We appreciate your feedback and would like to address your questions.

Missing examples

Thank you for your great suggestion! Following your advice, we have included two new figures in the pdf file attached to our general response. One is to illustrate FactorSim with a concrete example, and the other will explain our robotics experiment in more detail. See more detail below.

The connection between FactorSim and the formulations

While LLMs are free to generate code, our formulation in the paper (i.e., Algorithm 1) directly informs what is included in the prompt as context. To clearly illustrate how FactorSim's prompts are constructed and map to the theory presented in Algorithm 1, we made a new figure and attached it to the PDF of our general response. FactorSim uses five prompts, as highlighted in grey boxes in the newly added figure. The first prompt corresponds to Equation 1 in our main paper. It decomposes the long and complex simulation specification into a list of steps to be implemented sequentially. The output from that is a list of instructions. For example, a step instruction could be "Introduce a blue agent rendered as a blue circle, allowing players to control it with arrow keys. The red puck should respawn when it collides with the blue agent." Then, at each step, FactorSim would add and modify the code of the existing codebase to incorporate the instructed change.

During each step, FactorSim uses the second prompt (i.e. Context Selection Prompt, as shown in the figure) to select a set of state variables from the list of all state variables in the current codebase. This step corresponds to Equations 9 and 10 in Algorithm 1 in our main paper. The output of this prompt consists of a list of state variables, including state variables that are considered to be relevant to this module and new state variables that LLMs deem necessary to add to the system. After the list of context states is obtained, we will use it to retrieve the relevant functions in the codebase. A function is defined as relevant to a state variable if the state variable is used in the function. Our factorized POMDP representation essentially arranges a codebase into a (hyper-)graph where state variables are nodes and functions are hyperedges. After we obtain the list of relevant states and functions, they are used as context for the existing codebase in the following prompts the LLMs to generate the "model," "view," and "controller" part of the module. Note that not all steps require all three components; for instance, in our robotics experiment, only the "model" is generated as the "view" and "controller" are determined by the robot and the physics engine.

Clarification on our robotics experiment

The paper GenSim proposed a benchmark to evaluate how well LLMs can design and implement simulation tasks, as well as improve their performance in task generation. We follow the same experimental setup described in their paper, with the addition of a new metric—"human pass rate"—to assess prompt alignment.

In our experiments, we apply FactorSim to the benchmark proposed by the paper GenSim [2]. The benchmark, built on top of CLIPort [3], is designed to investigate how well Large Language Models (LLMs) can design and implement simulation tasks and how to improve their performance on task generation. This benchmark includes a list of tasks described in the following format:

• task-name: build-car
• task-description: Construct a simple car structure using blocks and cylinders.
• assets-used: [block/block.urdf, ball/ball-template.urdf]

The assets needed are provided in the prompts. The robot control and camera parameters are assumed to be given. Refer to Figure 2 in the attached PDF for an overview of our robotics experiment. We apply FactorSim to generate robotics tasks in a very similar way to how we generate RL games, as both can be modeled as POMDPs.

To further demonstrate FactorSim's effectiveness in generating robotics tasks, we curate a list of 12 assembly-related tasks and show that FactorSim outperforms all Chain-of-Thought baselines used in GenSim by a large margin due to its ability to factorize the multi-step process into subtasks and generate these subtasks in separate steps, by using only the relevant context needed. Due to space limitations, please find more details in our response to reviewer 6Px5.

Unclear input mapping

In our zero-shot transfer experiment, we train RL agents on generated code and test them on the original game implementation in the Pygame Learning Environment benchmark (PLE) [1]. In the Pygame Learning Environment (PLE), an action is defined as a "keyboard button." When we say, "The same button always means the same thing," we mean that a specific keyboard key consistently maps to the same in-game action during RL agent training. For example, pressing the spacebar in Flappy Bird will always trigger the flap action in training and testing environments. We will rewrite the sentence in our paper for better clarity. Thank you for pointing it out!

Reproducibility

We are committed to releasing the code publicly. We will ensure that the code repository includes comprehensive documentation and examples to facilitate the reproduction of our results. We have sent our code, with instructions on how to run it, via an anonymous link to the AC, following the conference guidelines.

References

[1] Tasfi, Norman. "Pygame learning environment." GitHub repository (2016).

[2] Wang, Lirui, et al. "Gensim: Generating robotic simulation tasks via large language models." ICLR 2024

[3] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Cliport: What and where pathways for robotic manipulation." Conference on robot learning. PMLR, 2022.