SoftPhy: Soft-Body Physical Concept Learning and Reasoning from Videos

2. Former works are in support of this combination. Concepts such as deformability and elasticity, which is highly abstract and on the different descriptional levels, have been equally proposed and discussed in a series of former cognitive AI research. For example, in Physion++([A]), it includes mass, deformability and elasticity and put them at the same level (see ref[A] Fig.1&2). The parameter levels discussed in our work are proposed similarly as Physion++([A]).

[A] Tung, Hsiao-Yu, et al. "Physion++: Evaluating physical scene understanding that requires online inference of different physical properties." NeurIPS (2023).

3. High level properties are easy to both understand and control. Using APIs from physics engine facilitates the control of diverse parameters and allows us to change different settings in different videos, while it would be inconvenient for us to modify the underlying parameters and to distribute them in a good manner(e.g. cloths are 50% likely to be very rough and 50% likely to be very smooth). Besides, we think that using APIs also assures that the concepts are familiar to humans, which tells why they are extracted as APIs.

[Q3(b). Question: How to measure high level properties in real world and align with the methods used in simulation?]

[P1S1] c. ...Besides, stretchiness and deformability are both parameters without corresponding basic physics meaning, which means you cannot measure them in the real world. How would the authors measure the stretchiness of cloth from the real world, as is the method of measurement coherent with what’s inside the physics engine?

1. These high-level concepts are valuable in cognitive AI research. As we mentioned above, in this work, we focus on those concepts acceptable for common people, like stretchiness, plasticity, etc. Just as you kindly stated, they are kind of subjective and can hardly be strictly defined in physics and measured in the real world. But as our human study shows, most people can distinguish and apply these physical concepts, which means these concepts are valuable to discuss in cognitive AI field. We hope models can understand these high-level concepts like humans.

2. It's worth noting that, in this work, we focus more on fuzzy judgment (e.g. which one's X is much greater), not accurate prediction. Just as you kindly pointed out, these concepts like stretchiness can hardly be strictly defined by any physics formula, so we choose to design fuzzy judgment tasks. And we think measuring the accurate values might not be an urgent problem. In detail, the physical parameter values in the sampling pool are not diversely specified. Typically, for most parameters, we only set 2 or 3 values to sample from and they are greatly different. For example, for a cloth's stretching stiffness, we set 2 values, one value very high and the other very low, to make the cloth appear either difficult to stretch or highly elastic, so that the property difference can be easily captured by humans at the first glance. Thus, in this work, we focus on the agent's ability to distinguish properties, rather than predict numerical values. At least at this visual-language AI research stage, we believe if models fail to do simple fuzzy judgment in the simulation dataset, they may hardly predict the values as well, regardless of whether it is simulated or real-world.

3. Although the properties are not measured from simulation (but preset values), we can still find the closest physics parameters and measure them using the same method both in simulation and real world. Take stretchiness as an example, we think people may agree that it shares a very close meaning with Young's modulus which has strict physical definition. So if we need to collect data in the real world, maybe we can take Young's modulus as a criteria of stretchiness. In mechanics of materials,

   • Formula of Young's Modulus: $E = \frac{\sigma}{\varepsilon}$

     • Where $E$ is Young's Modulus, $\sigma$ is stress, and $\varepsilon$ is strain

   • Stress ($\sigma$): $\sigma = \frac{F}{A}$

     • Where $\sigma$ is stress, $F$ is the force applied, and $A$ is the area of the surface on which the force is acting.

   • Strain ($\varepsilon$): $\varepsilon = \frac{\Delta L}{L_0}$

     • Where $\varepsilon$ is strain, $\Delta L$ is the change in length, and $L_0$ is the original length.

   Since in the dataset we record the 2D/3D data (image, segmentation, point cloud, mesh, etc.) for each frame, we can follow the above formulas to measure the property value just like scientists do in the real world.

Thank you for the constructive comments!

Q1(a). About Pulley

[P1S1] a. In Intro – Second Paragraph. The examples given for humans are very irrelevant to the “soft body”, the topic of this paper. For the liquid example, it can demonstrate the density, but for the pulley example, which physics parameters are the humans trying to distinguish?

We are sorry for the potential confusion regarding the liquid and rope (pulley) example. Generally, in this work, we emphasize both the parametric side and the behavioral side of the soft-bodies. In detail, we address this question from two perspectives.

[Why we choose pulley system as the basis of rope scenario?] We believe, as for human perception, rope is very different from liquid in that rope is 1-dimensional. When we think of rope, we mainly consider the connectivity and length constraints in ropes as well as the derivative dynamics. Viewing liquid dynamics is quite different when we may focus more on the physical properties and its global contour behaviors. In the rope(aka. pulley) dataset, we attempt to guide models to learn the embedded physical mechanics and constraints behind a variety of objects. To some extent, lift-up and push-down, cooperating with pulleys, are the most common use of rope in both everyday life(e.g. gym) and industry(e.g. crane). For a simple case, humans could easily perceive the motion of another side of the rope if we see one side of the rope lifted up or pushed down, which is proven diffcult for many VLMs due to dataset gap. As for a more complex case, we know that the dynamic pulley in Fig. 1 (c) of the paper could reduce the tension between ropes compared with static pulley, helping us to pull the cargo up more easily. Hence, we think it's crucial to equip vision language models with a conceptual perception of these phenomena, which is more related to physical dynamics reasoning, as opposed to physical parameter or concept reasoning. For each scenario, we consciously include both reasoning sides.

[What physical parameters relevant to "soft-body" we are trying to distinguish?] Typical physical parameters of rope are like elastic coefficient and bending stiffness. Since these parameters are already displayed in the scenario of cloth or ball, where the parameters are shown in a more comprehensive and complicated way, thus considering the diversity of physical parameters in our overall dataset, in the rope scenario, we mainly focus on other physical characteristics of this kind of material. One is the relative weight of load attached to the rope, indicating the external force exerted to the rope terminals, the other is the relative tension force within the rope segments. We hope these two physical parameters can be ideal testing criteria for the conceptual/parametric perception of rope.

Q1(b). About Logic Leap

[P1S1] Moreover, humans can distinguish these physics properties do no means AI models can, so there is a logic leap between the human examples to “However, it remains an …”

Thank you for your concern. In this paragraph, we express concerns about whether current AI models have the capability to accurately identify distinctions in the physical properties and dynamics of soft-body objects, which humans can intuitively discern. To improve our writing, we show the revised paragraph below and have incorporated it in our revision of paper.

We are able to know that the clear liquid in Fig.1(a) at the bottom has a higher density than the yellow liquid on the top; we know that the dynamic pulley in Fig.1(c) could reduce the tension between ropes compared with static pulley, helping us to pull the cargo up more easily. These innate human skills raise an intriguing question：can current AI models have the physical common sense to infer soft objects' physical properties and predict their corresponding dynamics?

Q2. About Definition of Soft-Body

[P1S1] b. From this paragraph on, I notice that the paper has a confusing meaning of “soft body”, how is liquid a type of soft body? According to Wikipedia (Soft-body dynamics), it should be a solid object at least. Yes, I can find more rigorous sources (e.g. a textbook), but I think a simple checkup on Wikipedia can avoid such concept mistakes.

The Wikipedia focuses on Soft-body dynamics, a specialized area within computer graphics. This term is used to differentiate soft-body dynamics from fluid dynamics, distinct in computer graphics simulation process. While our emphasis is on physical reasoning, not graphic methods. We adopt this concept to underline that prior benchmarks primarily featured rigid bodies, like spheres and cubes. Fluids, being neither rigid nor adequately addressed previously, are a separate and worthwile consideration.

We hope our clarification can resolve your concerns on the "soft body". If there are still concerns on the confusion of the name or any suggestion, we can discuss more and revise the whole paper correspondingly.

During rebuttal, the authors have made clear the scope of the "physics" in this work. Given this change of precondition, I can relax the questions on the rigor of physics. But it still does not justify including liquid objects in soft objects, and I see no possibility to fix this. Thus, I raise my rating to 5.

Thank you for your insightful suggestions! We agree with your idea that physical concepts should be clearer and more rigorous. Therefore, we've broadened our work's description from 'soft body' to 'continuum' to more accurately reflect its scope.

In both physics and computer graphics, 'continuum' encompasses various bodies such as liquids, soft materials (e.g., soft balls, cloth, and ropes), rigid bodies (e.g., cubes, pillars, plates, and spheres), and articulated bodies (e.g., pulleys). Our dataset comprehensively encompasses fluids, soft bodies, rigid bodies, and articulated bodies (e.g., in the rope scenario). This inclusion leads to our utilization of the continuum concept, enhancing the breadth and relevance of our study.

Moreover, we consciously include both physical dynamics reasoning (e.g., interactions between fluids, soft bodies and rigid bodies), and physical parameter or concept reasoning (e.g., density for fluids; tension, elasticity for soft bodies; mass for rigid bodies). For instance, our rope and pulley scenario involves elements of rope, rigid bodies, and articulated bodies; the fluid scenario includes liquids; the cloth scenario covers both cloth and rigid bodies; and the ball scenario focuses on soft balls. This extensive coverage ensures that our dataset provides a comprehensive understanding of the interactions and couplings within these various types of continua, capturing the complexity and diversity of real-world physical phenomena.

In light of this, we have renamed the dataset from SoPhy to ContPhy. We have added an entire section in Appendix A.4 to elaborate on the concept of the continuum. We have also updated the entire paper accordingly.

We hope this clarification addresses all your concerns. We are looking forward to your further reply and suggestions. If you find our response satisfactory, we would be grateful if you could consider revising your score.

Thank you for the constructive comments!

We are glad to hear that you think our SOPHY dataset is novel and invaluable, which advances ML / AI techniques to bridge the gap between human and AI in the physical world.

We are looking forward for your further feedback!

Thank you for all these constructive comments!

Q1. About Further Discussion of Experiments

Thank you for all these suggestions! We discuss each question below, and have added them in Section 4.2 of our revised paper.

[Vp2g] In Section 4.2 Paragraph "Physical Property", The author stated that "ALPRO achieves the best results in the rope scenario, and maintains competitive results in other scenarios, showing the value of large-scale video-text pre-training and alignment.". However, why other models slightly outperform ALPRO in scenarios other than Rope is not discussed.

[Q1 (a). Other Models Slightly Outperform ALPRO] Actually, all the baseline models struggle to achieve decent performance on physical property questions, except ALPRO, which achieves the best results in the rope scenario and maintains competitive results in other scenarios, showing the advantages of large-scale video-text pre-training and alignment.

[Vp2g] In Section 4.2 Paragraph "Dynamics", only the result of ALPRO and HCRN is discussed, and why other models do not work well is missing.

[Q1 (b). Other Baselines] Traditional pure neural networks have difficulty understanding the physical scenarios and in capturing the physical laws from videos and question-answer pairs. Thus, they perform worse than their previous performance on our benchmark.

[Vp2g] In Section 4.2 Paragraph "Scenario Analysis.", only cloth and rope scenarios are discussed.

[Q1 $c$. Other Scenarios] ALPRO performs well in the rope and cloth scenario. Except for the reason that the cloth and rope scenarios share some similarities, another important reason is the fewer question types in the rope and cloth scenario than those in the fluid and ball scenarios. Specifically, the rope scenario has counterfactual and goal-driven, and the cloth scenario has predictive. Conversely, in the fluid and ball scenarios, we incorporated all four problem types, thereby making situations much more complicated. To effectively address these scenarios, models must tackle four distinct question types, each focusing different aspects of physical dynamics. Consequently, no baseline models can gain an absolute advantage in these scenarios. This indicates that our four proposed question types well evaluate different dimensions of physical reasoning, making the fluid and ball scenarios particularly challenging for AI models.

All these comments have been updated in Section 4.2 of our revised paper to further analyze the experiments.

Q2. Reference of Table

[Vp2g] In Section 4.2 Paragraph 1, "We summarize the performance of all baselines in Table 1.", should be Table 2.

Thanks for the reminder. We have already fixed this and checked the whole paper again to improve our writing.

Q3. About Concerns for Semantic Information

[Vp2g] In Section 4.2 Paragraph "Evaluation Conclusion", the authors concluded that "Machine models results show that even state-of-the-art models struggle with answering physical questions based on the visual input.". However, the relationship between "soft body physics reasoning capability" and "answering physical questions based on visual input" is not clear. For example, AI models may understand soft body physics well, but unable to understand the questions, as the semantic information is not recognized by the model.

Admittedly, semantic information can influence results. However, using QA formats to evaluate physical reasoning abilities has been widely acknowledged and shown effectiveness in previous works, such as ComPhy [A] and CLEVRER [B]. QA gives the flexibility to estimate different kinds of physical reasoning abilities such as understanding different physical properties like mass, density, viscosity, and predict the corresponding dynamics based on variance of such physical properties. To correctly answer these questions, AI models need to have both the physical common sense and the ability to understand natural language queries and answers. Since the question-answer pairs in our benchmark are synthesized by templates, the major challenge of AI models is from physical reasoning. Thus, we claim that our SoPhy benchmark mainly evaluates "soft-body physical reasoning capability".

[A] Chen, Zhenfang, et al. "ComPhy: Compositional physical reasoning of objects and events from videos." NeurIPS (2022).

[B] Yi, Kexin, et al. "CLEVRER: Collision events for video representation and reasoning." ICLR (2020).

Thank you for all these constructive comments!

Q1. About Extending the Framework for Other Tasks

[t1H7] the task family is limited to the designed four types. Also the questions are generated from pre-defined sets of templates. These restrict the general use of the benchmark for other tasks, environments, and questions. Could the authors comment on how is it possible to extend the framework for other tasks?

We have developed various templates for each question type across four distinct scenarios. Our experiments reveal that current AI models fail in understanding soft-body physical dynamics, leading to poor performance on our dataset, independent of template design. Additionally, our dataset generation framework is capable of producing different question types and video types, such video annotations for segmentation and depth. This framework can support a range of common vision tasks, such as video segmentation, object detection, text-video retrieval and video grounding. Since our primary objective is to evaluate physical reasoning abilities in the form of question answering, we will not prioritize other vision tasks.

Q2. About Previous Benchmarks

[t1H7] the authors claimed that previous benchmarks cannot change mass and friction, but as many of them are also based on physical simulators, it's unclear why they couldn't do that.

We agree with the reviewer that many prior benchmarks, such as Physion [A] and ComPhy [B], are also based on physical simulators. However, their benchmarks fall short in thoroughly assessing whether machine models have human-like visual reasoning abilities, particularly in understanding physical object properties and dynamics. These physical benchmarks do not adequately cover dynamics influenced by varying physical object properties across diverse scenarios, notably in the case of soft objects. Some of them lack the variance of object parameters, such as Physion [A], in which solids share the same mass and water maintains a consistent density. Some of them limit their scope to simple primitives and scenario, such as ComPhy [B], in which they only consider mass and electric charge in a basic solid collision scenario of cubes and spheres. In contrast, we are the only comprehensive benchmark with object-specific properties in a wide range of scenarios, which even encompasses soft-body objects and unique language-based questions about dynamics in counterfactual and goal-planning scenarios. Our dataset addresses the gap in existing benchmarks, which often overlook complex intrinsic properties and corresponding dynamics in diverse scenarios, a key aspect our SoPhy emphasizes.

[A] Bear, Daniel M., et al. "Physion: Evaluating physical prediction from vision in humans and machines." NeurIPS (2023).

[B] Chen, Zhenfang, et al. "ComPhy: Compositional physical reasoning of objects and events from videos." NeurIPS (2022).

Q3. About Proposed Solution

[t1H7] the paper doesn't propose a solution to improve the performance based on the findings.

We thank the reviewer for this suggestion. We have presented a proposed solution for the fluid scenairo to better investigate the characterstics of the current benchmark in G2 of the General Response. Please kindly refer to it and we are looking forward to your further feedbacks!

[G2(d). Physical Simulation Model] Previous methods [A, K, L] mainly adopt DPI-Net as physical simulator, which is a 3D particle-based graph neural network [G]. DPI-Net performs well in mass [L], while falls short in fluid dynamics prediction. Therefore, we utilize a differentiable MPM physics simulation module DiffMPM [N] to provide a primary solution to the system identification of the fluid scenario. We also borrowed the simulation framework partly from PAC-NeRF [H].

[A] Chen, Zhenfang, et al. "ComPhy: Compositional physical reasoning of objects and events from videos." NeurIPS (2022).

[G] Li, Yunzhu, et al. "Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids." ICLR (2019).

[H] Li, Xuan, et al. "PAC-NeRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification." ICLR (2023).

[K] Bear, Daniel M., et al. "Physion: Evaluating physical prediction from vision in humans and machines." NeurIPS (2023).

[L] Tung, Hsiao-Yu, et al. "Physion++: Evaluating physical scene understanding that requires online inference of different physical properties." NeurIPS (2023).

[N] Hu, Yuanming, et al. "DiffTaichi: Differentiable programming for physical simulation." ICLR (2020).

[G2(e). Result Analysis]

The physical simulation method [H] in oracle model exhibits powerful performance that can precisely simulate soft-body dynamics, while it requires scenario-specific manipulation of hyperparameters. Therefore, we present representative results in the fluid scenarios below, which encompasses all four question types, and compare them with multiple baseline models. We report the representative results in the table above, which reveals that our SoPRO successfully outperforms all questions in the fluid scenario.

In property questions, SoPRO impresses with a 78.0% overall accuracy. This category includes two tasks: Stick Number which is much more simpler and Density which is more complex. The high accuracy in Stick Number significantly raises SoPRO's overall performance in property questions, making it a standout model. In dynamics questions, SoPRO surpasses all baselines, particularly in predictive questions, hitting a 68.3% accuracy rate. This is because simulator's objective focuses on predicting dynamics during its training.

[H] Li, Xuan, et al. "PAC-NeRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification." ICLR (2023).

[G2(f). Conclusion] We have proposed a new oracle model SoPRO, that marries physical-based dynamics models with large language models to reason on soft-body physical concept. We evaluate SoPRO in the fluid scenario and analyze comprehensively. The great performance of SoPRO-Oracle, with the initial input of cloud points that cannot be directly perceived, serves as a guide and oracle for future AI models.

Note that we only summarize each section in this comment. We have already incorporated the detailed method as a new section in Appendix A.3.

[G3. Revision Summary]

We made the following modifications in our revision to address reviewers' questions (highlighted in blue in the revision):

• Oracle model as our solution to this benchmark (t1H7)
• New nested pie charts with ratio numbers for better visualization (P1S1)
• New citation and discussion of paper [J] (Vp2g)
• Correct reference of Table 1 (Vp2g) and improved writing (P1S1)

[J] Li, Shiqian, et al. "On the Learning Mechanisms in Physical Reasoning." NeurIPS (2022).