OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

4. How does the method handle permanent deformation for viscoelastic and plastic materials?

The Material Point Method (MPM) can naturally handle permanent deformation for viscoelastic and plastic materials. Our method treats all deformations and materials in the same way.

5. Would there be a failure during training? For example, a bad parameter causes a 3D model to break down into pieces.

We understand the reviewer's concern. We addressed this issue by two strategies.

• We adjusted the simulator's hyperparameters to ensure the stability of the simulation. Therefore, the model is less likely to generate physically unrealistic results. Notably, we only observe the breakdown or explosion of a 3D object when the simulating fails, such as when the simulating grids are too coarse or the time step is too large.
• As mentioned in the manuscript, we utilize the multi-batch training strategy to deal with the optimization difficulty of MPM steps. Our method will optimize a stage multiple times from the same starting state. In this way, when non-physical results are generated in one stage, the diffusion guidance can correct them since the starting state is not changed.

6. More notes in Figure 3 can be added to illustrate how the decoder assigns different materials to different parts of the scene.

We appreciate the reviewer's suggestion. We have revised Figure 3 to include more notes to illustrate how the decoder assigns different materials to different parts of the scene. We presented the modified figure in Section D of the Rebuttal Appendix.

7. How is the shape of the water container specified in the wolf on the water scene?

We do not add any extra containers in that case. The reason why the water seems to be contained in a container is that the simulation is restricted to a $1\times1\times1$ cube. When initializing the particles, their positions are normalized to the range $[0, 1]$. During the simulation, we clamp the particles' positions to the range $[0, 1]$, which makes the water look like it is constrained in a container.

We will add all the aforementioned discussions to the revision of this manuscript.

[1] A Generalized Constitutive Model for Versatile MPM Simulation and Inverse Learning with Differentiable Physics. Proceedings of the ACM on Computer Graphics and Interactive Techniques (Symposium on Computer Animation), 2023.

[2] Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics. The International Conference on Machine Learning (ICML), 2023.

We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the novelty of our method, which first introduces learnable constitutive models in Gaussian Splatting and supports various materials with user-friendly text prompts. We are really delighted that you like our work and lean towards acceptance. Below, we provide clarifications for the concerns raised. Additional analysis, experiments, and visualization results are included in the supplementary material, where we provide a Rebuttal Appendix (rebuttal-appendix.pdf) and supplementary videos (rebuttal-videos). We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.

1. The method may lack robustness when dealing with unusual materials or scenarios since the diffusion guidance may be limited.

Our method leverages the video diffusion model to guide the prediction of material properties. We acknowledge that the diffusion guidance may struggle with unusual corner cases and complex scenarios, which is a common limitation of data-driven methods. Despite the limitations, we claim that other core components of our method, such as the physics-guided network and the MPM simulator, can be optimized directly with the ground-truth motion of unusual materials to reconstruct the dynamics of the scene. This may mitigate the limitation of the diffusion guidance under unusual scenarios.

2. Discussion on the related work A generalized constitutive model for versatile MPM simulation and inverse learning with differentiable physics [1]. What are the limits of the constitutive models? Would the method allow for a material mixture?

We appreciate the reviewer's suggestion. In the related work [1], the authors propose a generalized constitutive model, which is a linear combination of several predefined constitutive models. The coefficients of the linear combination are learnable parameters. Therefore, the model can perform inverse learning to predict the coefficients from the observed motion. By adjusting the coefficients, they can simulate the materials mixture.

The main difference and contribution, in terms of the generalized constitutive model, of our work compared to [2] and [3] are two-fold:

• While [1] assumes a homogeneous scene (although mixed, the scene is still homogeneous based on a single mixed material), our method can handle heterogeneous scenes with multiple objects and materials, offering more flexibility.
• Our method designs a physics-guided network to predict the material properties of each particle, which is more expressive than simply utilizing a set of coefficients. In easier tasks, such as inverse learning where the ground-truth motion is given, the coefficients can be optimized well as shown in [1]. However, our experiments in Section C of the Rebuttal Appendix prove that simply optimizing a probability vector (i.e., the coefficients) is hard to converge under our task setting.

The constitutive model can capture various material properties, but they probably cannot perfectly conform to real-world physics. Meanwhile, off-the-shelf constitutive models are limited to several representative materials, which may not cover all the materials in the real world. Despite the limitations, we believe that the constitutive models can provide a good approximation of real-world physics and greatly enhance the physical plausibility of the generated dynamics.

Our method can handle material mixture by simply removing the argmax operation in the physical-aware decoder and using the softmax output as linear coefficients for the constitutive models. However, in this work, we choose to assign a single material instead of a mixture of materials for each neighborhood. We claim that simply performing a linear combination of outputs of constitutive models may violate the original physical meaning of the constitutive models, which can lead to non-physical results and a lack of interpretability.

We acknowledge that we may overlook some important works and would appreciate any suggestions.

3. How are the parameters being chosen?

We choose the simulator's hyperparameters such that the simulation is stable and the results are visually plausible. The specific choices of the parameters are based on our empirical experience without a large-scale search. Other hyperparameters, such as the learning rate and the number of training iterations, are also chosen based on our empirical experience. We find that the model is not sensitive to these hyperparameters.

We initialize the weights of our neural network randomly (using the default initialization in PyTorch). The physical parameters of the particles are initialized to be the same for all particles in the scene. We choose the initial Young's modulus to be $2\times10^6$ and the Poisson's ratio to be $0.3$ according to NCLaw [2].

Thank you for taking the time to review our submission. We would like to kindly remind you to share any additional feedback or comments if possible. Your insights would greatly help us address any remaining concerns and further improve our work. We deeply appreciate your time and efforts.

We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the novelty and effectiveness of our method, and the breadth of the experiments, and that you found our manuscript well-written and easy to understand. We are truly delighted by your support and your inclination towards acceptance. Below, we provide clarifications for the concerns raised. Additional analysis, experiments, and visualization results are included in the supplementary material, where we provide a Rebuttal Appendix (rebuttal-appendix.pdf) and supplementary videos (rebuttal-videos). We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.

1. Figure 1 is confusing because the mountain is depicted as non-elastic but collapses after the duck falls.

We are sorry for the confusion caused by the example in Figure 1. The mountain is expected to be similar to a pile of plastic, deformable sand or mud. We depicted the mountain as non-elastic since its deformation is permanent and it does not recover its original shape after the deformation. In contrast, an elastic material would recover its original shape, such as the rubber duck in the same scene. Therefore, we think that the collapse of the mountain is consistent with the assigned material properties.

2. The interactions between a single object and an entire scene.

We appreciate the reviewer's suggestion. We have conducted experiments on real-world scenes, including the flower vase scene and the fox scene. Following PhysGaussian [1], a desired area of the scene is simulated and optimized to match the text prompt. We provide visualization results in Section E of the Rebuttal Appendix and supplementary videos. We hope that these experiments can demonstrate the effectiveness of our method in modeling the interactions between a single object and an entire scene.

3. The performance in scenarios involving more than two objects.

We appreciate the reviewer's suggestion. We have conducted experiments on scenes involving more than two objects, including the pillow-basket scene and the material mixture scene. Despite the memory-efficient MPM solver, the increased number of objects requires more GPU memory and computation, which may limit the complexity of the scene. Implementing a distributed version of the MPM solver is a potential future direction. Meanwhile, although the model can output desirable dynamics for each object in our experiments, we claim that it would be more challenging to perform fine-grained control over the interactions between multiple objects. We provide visualization results in Section E of the Rebuttal Appendix and supplementary videos.

We will add all the aforementioned discussions to the revision of this manuscript.

[1] PhysGaussian: Physics-integrated 3d Gaussians for generative dynamics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

4. The technical design of dividing an object into small particle groups and allowing them to have different predefined constitutive models may not align with real-world physics.

The intuition behind our design is that real-world scenes are usually composed of multiple materials. Therefore, dividing the scene into small particle groups enables the model to assign different material properties to different parts of the scene according to the text prompt and the semantic or positional information of the particles. We understand the concerns when a single object is divided into multiple parts with different constitutive models, which may result in inconsistent behaviors of the object. During our experiments, we found that the overall physical properties of a single object are determined by the majority of the particles, which provides a reasonable approximation of real-world physics and enhances the flexibility of our method. We also find that our model can easily converge to a homogeneous material in a single-object scene (Section C of the Rebuttal Appendix).

5. Why a neural network is needed? What's the advantage of a neural network compared to optimizing a one-hot softmax vector for classification?

In our early experiments, we tried optimizing a vector representing the probability distribution over different constitutive models for each Gaussian particle. However, we found that this simple method was really difficult to converge and prone to numerical instability. In contrast, the neural network can effectively extract the features of the scene and utilize the neighborhood information of the particles to predict the material properties. We provide visualization comparison results of training our neural network and optimizing a one-hot softmax vector in Section C of the Rebuttal Appendix, where the neural network achieves better performance in terms of convergence and stability.

6. The notation of physical parameters.

We are sorry for the confusion caused by the notation of physical parameters. The physical parameter itself is a real scalar value, but a particle may have multiple kinds of physical parameters, such as Young's modulus and Poisson's ratio. Therefore, it would be better to use a vector to represent the physical parameters of a particle, i.e., $\boldsymbol{\gamma}\in\mathbb{R}^K$, where $K$ is the number of different physical parameters. We will revise the notation in the manuscript to make it more clear.

We will add all the aforementioned discussions to the revision of this manuscript.

3. The CLIP metrics may be unconvincing. It's preferred to judge motion realism by human preference.

We appreciate the reviewer's suggestion. We agree that CLIP metrics may lack the ability to evaluate motion realism since the metrics are calculated on individual frames. As a complement to the metrics, we conducted a user study among 20 participants to evaluate the quality of the generated dynamics by different methods. During the study, the participants were asked to rank different videos based on both the text alignment and physical plausibility of the dynamics. The following table shows the detailed results of the user study, where the numbers represent the average ranking of each method. The lower the number, the better the performance.

Single Object

Multiple Objects

The results indicate that our method achieves better performance in modeling various kinds of materials. Specifically, the baselines achieve close performance to that of our method in modeling single pure elastic objects, but they struggle to model the behaviors of other materials (e.g., plasticity, viscoelasticity, fluid) especially when the scene is composed of multiple materials. This conclusion is consistent with the quantitative and qualitative results in Section 4.2 of the manuscript. We provide our user interface and better visualization of the table in Section A of the Rebuttal Appendix.

We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the significance of our target problem, as well as the flexibility and effectiveness of our method, which overcomes the limitations of existing approaches. We are also pleased that you found our memory-efficient MPM solver useful and the design of our experiments interesting. Below, we provide clarifications for the concerns raised. Additional analysis, experiments, and visualization results are included in the supplementary material, where we provide a Rebuttal Appendix (rebuttal-appendix.pdf) and supplementary videos (rebuttal-videos). We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.

1. No real-world experiments.

We appreciate the reviewer's suggestion. We have conducted experiments on real-world datasets, including the flower vase scene and the fox scene. We provide visualization results in Section E of the Rebuttal Appendix and supplementary videos. We hope that these experiments can demonstrate the effectiveness of our method in real-world scenarios.

2. The experiment result of the can-duck scene is weird.

We appreciate the reviewer's observation. By analyzing our collision experiments (e.g., the can-duck scene), we found that the unrealistic motion is caused by the choice of grid resolution used in simulations. The Material Point Method (MPM) utilizes grids to gather information from particles and subsequently transfer the information back to the particles. Consequently, using a lower grid resolution may lead to inadequate physical contact during collisions. This occurs because particles within a large grid may influence one another even when they are not in direct contact. In the original experiments, we used $25\times25\times25$ grids for all scenes for a fair comparison. Although this number is sufficient for most cases, low-resolution grids may cause artifacts such as collision without physical contact. Given the reviewer's feedback, we rerun the experiment with higher grid resolutions for the collision scene and achieve more realistic collision results. We provide analysis and visualization results in Section B of the Rebuttal Appendix and supplementary videos.

Thank you for taking the time to review our submission. We would like to kindly remind you to share any additional feedback or comments if possible. Your insights would greatly help us address any remaining concerns and further improve our work. We deeply appreciate your time and efforts.

We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you appreciate our work, which incorporates 3D Constitutive Gaussians, integrates domain-expert constitutive models, and employs effective training strategies for learning material properties in dynamic generation. Below, we provide clarifications for the concerns raised. Additional analysis, experiments, and visualization results are included in the supplementary material, where we provide a Rebuttal Appendix (rebuttal-appendix.pdf) and supplementary videos (rebuttal-videos).
We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.

1. The comparisons are limited to the same object with different materials. More experiments on controlling the physical parameters of the same object are needed.

We appreciate the reviewer's suggestion. Our method is capable of controlling the physical parameters of the same object. To demonstrate the flexibility of our method in controlling the physical strength, such as the softness and hardness, of the same object, we conducted experiments including the ficus scene, the wolf scene, the jelly scene, and the material mixture scene. We provide visualization results in Section E of the Rebuttal Appendix and supplementary videos.

2. Altering materials in PhysGaussian [1] is straightforward. This simplicity contrasts sharply with the complexity of optimizing a model using video diffusion models for similar tasks.

We agree that altering materials in PhysGaussian [1] is straightforward. This simplicity is because their experiments are limited to a single, homogeneous object/scene. In contrast, our method is designed to handle more complex scenarios, such as scenes with multiple objects and different materials. In this case, assigning appropriate material properties to each object is challenging and our method provides a flexible and effective solution.

3. The collision experiments appear to lack physical contact, whereas collisions modeled in PhysGaussian [1] are depicted as more substantial.

We appreciate the reviewer's observation. By analyzing our collision experiments (e.g., the can-duck scene), we found that the unrealistic motion is caused by the choice of grid resolution used in simulations. The Material Point Method (MPM) utilizes grids to gather information from particles and subsequently transfer the information back to the particles. Consequently, using a lower grid resolution may lead to inadequate physical contact during collisions. This occurs because particles within a large grid may influence one another even when they are not in direct contact. In the original experiments, we used $25\times25\times25$ grids for all scenes for a fair comparison. Although this number is sufficient for most cases, low-resolution grids may cause artifacts such as collision without physical contact. Given the reviewer's feedback, we rerun the experiment with higher grid resolutions for the collision scene and achieve more realistic collision results. We provide analysis and visualization results in Section B of the Rebuttal Appendix and supplementary videos.

We will add all the aforementioned discussions to the revision of this manuscript.

[1] PhysGaussian: Physics-integrated 3d Gaussians for generative dynamics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Thank you for taking the time to review our submission. We would like to kindly remind you to share any additional feedback or comments if possible. Your insights would greatly help us address any remaining concerns and further improve our work. We deeply appreciate your time and efforts.

I still have doubts about the second point, that even for multiple objects, assigning material types is not complex. And the examples are not actually complex scenes in my view. Also, I agree with review ktNn that there are flaws in the paper and more convincing results are needed. However, since the authors provided additional results for the different parameters for the same material properties and resolved the collision problem, and considering the pytorch implementation of MPM, I raised my score to 6.

4. Warp can create arrays from Torch tensor without copying. Why does Warp consume much more memory than implementing MPM in Pytorch?

Although WARP can create arrays from PyTorch tensors without copying, it automatically creates zero-gradient arrays and other intermediate arrays when the gradient is required. These additional arrays are not managed by PyTorch and can consume a substantial amount of extra memory due to the numerous MPM simulation steps involved. Removing these intermediate arrays is non-trivial, requiring significant modifications to the WARP library according to our early experiments. Therefore, previous work like PhysDreamer [1] had to employ KMeans clustering to reduce the number of particles used in simulations.

Our implementation of MPM in PyTorch takes advantage of the advanced memory management tools available in the framework, such as half-precision training and gradient checkpointing. In this work, we utilize gradient checkpointing as an effective compromise between memory usage and computational efficiency. Additionally, our implementation minimizes the communication overhead between PyTorch and WARP, resulting in reduced testing time. We hope that our PyTorch-based MPM implementation will serve as a valuable tool for future research.

5. More related works.

We appreciate the reviewer's suggestion. In the related work, Mixing Sauces: A Viscosity Blending Model for Shear Thinning Fluids [2] and A Generalized Constitutive Model for Versatile MPM Simulation and Inverse Learning with Differentiable Physics [3], the authors propose a generalized constitutive model. Specifically, [2] proposes to blend different constitutive models in a non-linear way and [3] proposes a linear combination of several predefined constitutive models. The weights or coefficients of each predefined constitutive model are learnable parameters. Therefore, the model can perform inverse learning to predict the coefficients from the observed motion.

The main difference and contribution, in terms of the generalized constitutive model, of our work compared to [2] and [3] are two-fold:

• While [2] and [3] assume a homogeneous scene (although mixed, the scene is still homogeneous based on a single mixed material), our method can handle heterogeneous scenes with multiple objects and materials, offering more flexibility.
• Our method designs a physics-guided network to predict the material properties of each particle, which is more expressive than simply utilizing a set of coefficients. In easier tasks, such as inverse learning where the ground-truth motion is given, the coefficients can be optimized well as shown in [2] and [3]. However, our experiments in Section C of the Rebuttal Appendix prove that simply optimizing a probability vector (i.e., the coefficients) is hard to converge under our task setting.

We acknowledge that we may overlook some important works and would appreciate any suggestions.

We will add all the aforementioned discussions to the revision of this manuscript.

[1] PhysDreamer: Physics-based interaction with 3d objects via video generation. European Conference on Computer Vision (ECCV), 2024.

[2] Mixing Sauces: A Viscosity Blending Model for Shear Thinning Fluids. ACM Transactions on Graphics (TOG), 2019.

[3] A Generalized Constitutive Model for Versatile MPM Simulation and Inverse Learning with Differentiable Physics. Proceedings of the ACM on Computer Graphics and Interactive Techniques (Symposium on Computer Animation), 2023.

We sincerely appreciate the reviewer's insightful and valuable feedback. We are encouraged to know that you recognize the novelty of our proposed learnable Constitutive Gaussians, the thoroughness of our comparisons and ablations, and that you find this work well-presented. Below, we provide clarifications for the concerns raised. Additional analysis, experiments, and visualization results are included in the supplementary material, where we provide a Rebuttal Appendix (rebuttal-appendix.pdf) and supplementary videos (rebuttal-videos). We greatly value your time and effort, and we welcome any follow-up questions or suggestions you may have.

1. How is the physical-aware decoder trained given that the discrete material class is non-differentiable?

The physical-aware decoder is trained using a differentiable approximation of the discrete material class. Specifically, the direct output of the physical-aware decoder is a continuous vector of shape (num_particles, num_materials), which is then passed through a softmax function to obtain a probability distribution over the materials. We choose the material with the highest probability for each particle, i.e., argmax(softmax(output)). Since the argmax operation is non-differentiable, its gradient is estimated using the straight-through estimator, which is a common trick in training nondifferentiable operations like argmax. The whole process is defined in the following differentiable hard_softmax function:

def hard_softmax(logits: Tensor, dim: int) -> Tensor:
    y_soft = logits.softmax(dim=dim)
    index = y_soft.argmax(dim=dim, keepdim=True)
    y_hard = torch.zeros_like(y_soft).scatter_(dim=dim, index=index, value=1.0)
    ret = y_hard - y_soft.detach() + y_soft
    return ret

In the forward pass, hard_softmax behaves the same as an argmax operation, while in the backward pass, it behaves the same as a softmax operation.

2. The control of the scene, such as initial velocity and external forces, is not learnable parameters. The text descriptions must be crafted based on these known, predefined interactions.

In this work, we mainly focus on learning the material properties of a given scene, which includes constitutive models and physical parameters such as Young's modulus. This task is challenging because it is difficult to assign appropriate material properties to numerous particles, especially in scenes that involve multiple materials. The interactions between objects are implicitly governed by these material properties.

We do not learn the initial velocity and external forces since users can easily control these factors (e.g., by dragging an arrow in an interactive user interface). It is noteworthy that the simulator-based feature of our method allows generalizing the learned material properties to new scenes with different initial conditions, for which we have conducted experiments in Section 4.2 Motion Generalization of the manuscript.

In terms of text descriptions, although it would be better to describe the motion, only describing the material also works in our method since the material implicitly determines the dynamics. For example, the prompt ``a rubber bear'' implies the bear is bouncy and elastic. New experiments are provided in Section E of the Rebuttal Appendix, where input text descriptions do not contain any verbs.

3. In reality, objects cannot be both elastic and sandy. Given such heuristic blending of materials within a single object, the dynamics may conform to unrealistic motions encoded in the video diffusion model.

We apologize for the confusion caused by the ficus tree example in Figure 4, which can appear either elastic or sandy depending on the prompts used. However, this demonstration highlights the flexibility in generating diverse material properties and illustrates how text prompts can be used to control these materials effectively. We believe there are some objects that may look quite similar but have different material properties, such as a blue jelly cube and a water cube with the same shape (Figure 7 in the manuscript appendix). In this case, the material properties can be controlled by the text prompt.

We admit the ambiguity and potential artifacts introduced by the data-driven nature of the video diffusion model. Our method predicts material properties using the conditional probability distributions provided by this model. Since the video diffusion model is trained on large real-world datasets, it is expected to assign higher probabilities to material properties that are more realistic and text-consistent, thus enhancing the physical plausibility of our method.

Thank you for taking the time to review our submission. We would like to kindly remind you to share any additional feedback or comments if possible. Your insights would greatly help us address any remaining concerns and further improve our work. We deeply appreciate your time and efforts.