I-PHYRE: Interactive Physical Reasoning

We thank Reviewer PPjJ for constructive feedback. Below, we answer questions raised by PPjJ and point out several misunderstandings.

Generalize to unseen games from the basic split.

We further created another 10 games similar to the basic games by varying the object positions and angles. Agent's generalization performance is shown in the following table. Overall, the performance in these new games is slightly inferior to the original basic games but better than noisy games.

The training dataset of 20 games for basic split seems too small a dataset to lead to any meaningful generalization to complex splits.

We choose not to scale up I-PHYRE for the following reasons.

1. Prior evidence does not support large-scale learning approach: As experimental results from the original PHYRE show, existing methods that excel in within-template tasks still fall short in cross-template settings, despite the fact that the agent learns from a vast amount of variations/data.
2. Minimal yet complete system for probing the boundary of intuitive physics models: Indeed, one can scale up by introducing stochasticity into the game dynamics. However, naive scaling risks creating unsolvable games, given that the games demand precise timing. Critically, one of the central goals of this paper is to probe the boundary of the intuitive physics models. We argue the presented simulator is already a minimal yet complete system to achieve this goal.
3. Learning generalizable and compositional physics: Given the generalization results presented in works like PHYRE, we set our primary goal in I-PHYRE to be agents that can generalize to unseen scenarios from learning physical primitives and composing them with well-designed modeling rather than merely data-driven learning. This paradigm has been justified in other work, such as the schema network (ICML 17'), where the model learns from only a single setup and becomes generalizable to a wide range of variations.

The success rate of these baselines on full task.

Thank you for pointing it out. We list the success rates of the methods below and will update this table in revision.

Whether I-PHYRE expects emergence of compositional generalization by training on just the basic split games? If yes, have authors tried using recurrent policies?

We do believe that compositional and systematic generalization can emerge by learning from basic games, as evidenced and justified by schema network (ICML 17'), MLC (Nature 21'), and RCN (Science 17'). Note that we do not commit to either MLPs or recurrent networks but rather believe in careful model design.

To directly answer the reviewer's question, we indeed have tried recurrent networks to build a model-based RL agent. However, we observe no significant performance difference; see Appendix E. We conclude that the lack of physics modeling hinders learning helpful dynamics. We discuss potential ways of physics modeling in Appendix H. We will further discuss Lake and Baroni's work in revision.

How is the combined strategy baseline implemented and what is the action space of this baseline?

The combined baseline predicts the execution time for all actions, waits and executes the earliest action, and updates the execution time. The action space per time step is the same as other strategies, which are all the blocks.

Why are the combined strategy baseline trained only for < 100k step in Fig. 3 experiments?

The ones that have < 100k steps in Fig. 3 are SACs, including Inadvance, Combine, and Onthefly. This is because different RL algorithms are set with different numbers of steps per iteration implemented in Ray RLlib. SAC is set with 1 by default and A2C with 10 by default. However, the number of iterations remains the same across different algorithms.

Have authors tried using a on-the-fly version of GPT-4 baseline presented in the appendix?

In our preliminary study, we tried basic games using GPT-4 with a planning on-the-fly strategy. The GPT-4 solves none of them and almost always returns no action. Oftentimes, it regards a new frame as no change from the previous one, as there are only limited changes in locations in between.

Thank you addressing my concerns. I have read authors response to all reviews. I would recommend authors to add the success rate in the main paper and results for testing generalization to unseen variants of basic split games. In addition, it would be nice if authors add more details to describe the combined baseline in the main paper, the current description is unclear. It'd also be good to add details about hyperparameters like max steps chosen for RL training in appendix. As most of my concerns are addressed, I will update my rating to reflect the score.

We are grateful for your recognition of I-PHYRE's importance and address your concerns below:

Add a table

We will include a table in the revised version, space permitting.

The author notes the absence of 3D interactive environments as a limitation.

While we acknowledge the potential benefits of a 3D interactive benchmark, our focus on 2D is a deliberate step from PHYRE's design, emphasizing interactivity. We hypothesize that the main challenge in 3D relates to perception, and with advancements in the vision field, a robust physics reasoning method will be increasingly valuable.

What potential solutions or recommendations does the author suggest for this benchmark?

As outlined in Section 5.1, we propose focusing on physics modeling, multi-step interventions, and action timing to build more powerful physical reasoning agents. The schema network (ICLR 17') is a promising starting point, advocating for transparent environment modeling to improve optimization and planning.

Thank you for appreciating our clear motivation, novel design, and thorough experiments and analysis.

The visualization of Figure 1. is slightly hard to understand at first glance.

We acknowledge your feedback. In response, we will divide Figure 1 into two subfigures to more effectively illustrate the challenges of I-PHYRE and the creation of game splits.

Although the experiment was thorough, it can be that 40 games is a small number of games to anticipate strong generalization.

We appreciate your concern. We limited the scale of I-PHYRE for several reasons:

1. Prior evidence against large-scale learning: Despite a large dataset, methods excelling in within-template tasks struggle in cross-template settings (as seen in PHYRE).
2. Minimal yet complete system: Scaling up can introduce unsolvable games due to the need for precise timing. Our goal is to probe the boundary of intuitive physics models, and the current simulator effectively serves this purpose.
3. Learning generalizable and compositional physics: We focus on generalization from learning physical primitives and composing them, as supported by research like the schema network (ICML 17'), which generalizes from a single setup to a range of variations.

Related work

We will extend our discussion on related work in the revised manuscript.

More dominant model-based reinforcement learning.

We will explore recent methods in model-based reinforcement learning and integrate them into our discussion, following the World Model paradigm.

Thanks for your comments. I have read the response and most of my concerns are addressed. I will keep my current rating as 8.

We are glad that you treat our paper as solid with both good intuitive and empirical results and acknowledge the contribution and significance of I-PHYRE. We will discuss and compare our work with [1] and [2] in the revised version.

For your concerns:

Then making a claim that this difficulty is due to the "inherent complexities of these tasks" is, while not unbelievable, strong.

We acknowledge the challenge in analyzing the innate complexities of games. Our research shows these complexities by highlighting the sampling difficulty in identifying successful solutions in Appendix B, providing an approximate estimation of game difficulties. We will clarify this point in the revised version.

Failure reasons

We appreciate your comments! In the revised version, we will elaborate on the experimental results presented in Appendix G and strengthen our claims with empirical evidence and hypotheses.

On Figure 1

Thank you for your suggestions. In response:

1. We used larger arrows to illustrate relationships and will enlarge them further if space permits.
2. We will add additional annotations and coloring.
3. We changed the caption to "Wrong order leads to missing elimination timing for two balls" and referred readers to our supplementary demo video for a detailed explanation.

"Combined" planning strategy needs better explanation.

The plan updates after each action. For example, if the initial plan involves actions at times t1 and t2 (t1 < t2), the agent waits until t1, executes the action, and updates the plan for the following action.

On Figure 2

We will revise Figure 2 to:

1. Separate the bars according to strategies.
2. Depict human results as horizontal lines for direct comparison.

On Figure 3

We will organize Figure 3 into groups in the revised version.

Contribution statements

We will incorporate your suggestions in the revised manuscript.

Different naming conventions for agents

We will ensure consistency in agent naming throughout the paper.

Failure on I-PHYRE

We view multi-step reasoning and action timing as scientific problems, while delayed reward and sparse action are computational challenges. The multi-step nature causes delayed rewards, and action timing implies sparsity. We will distinguish hypotheses from claims in the revised version.

Nature of the "combined" strategy

The combined strategy, inspired by human thinking, maintains a global vector of pre-planned action timing, updating it intermittently. This approach balances between constant updates and pre-planning, re-planning only at key frames for computational efficiency and effectiveness.