NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics

Thank you for reviewing our work. Your feedback is instrumental in strengthening our paper. Here are our responses to your concerns.

W1: The dataset should be described in more detail. For example, the number of views to initialize the scene, and the number of views for the subsequent optimization. How many episodes are used to train each kind of material? These factors are important because they will tell readers the efficiency of the learning-based material law.

Thank you for your thoughtful consideration. Each scene from our synthetic data contains 50 uniformly sampled viewpoints for initial state estimation, with the cameras evenly spaced on a sphere covering the object of interest. In addition, we record 400 frames depicting the object’s subsequent motion from a single view. We use the first 20 frames from the dynamic observation to infer the object's initial velocity and the entire trajectories for material adaptor optimization. We train each material adaptor for 1,000 iterations.

W2: In Appendix C, the authors claim that they provide videos in supplementary materials, but I haven’t found them in the submission system.

We apologize for not managing to upload the video demonstrations of our synthetic dataset at submission. As a remedy, we supplement some key frames to show the object motion in Figure C in the attached PDF. We will make sure to release the synthetic data later.

W3: In Table 1, Fig. 3, and Fig. 4, the results show that the NeuMA overall outperforms the NeuMA w/ PS. I think this could be a bit weird. The groundtruth image sequences are generated by the the groundtruth 3D particle tracks. Thus, the information contained in the particle supervision should not be less than that in the 2D image supervision. However, the results shown by the authors illustrate that only using 2D image supervisions is better than using the 3D particle correspondence. I think the metrics of the model trained by the 3D labels should be the upper limitation of the methods and the result of 2D supervised methods should try to approximate it but cannot entirely surpass it so much. Therefore I think the author did not train the 3D baseline thoroughly, which may be due to the improper selection of training strategy or base model.

Thoughtful viewpoint! The mentioned results may seem weird at first glance but can actually be attributed to the following aspects.

• Texture information: Although the image sequences are generated from ground-truth particles, they are rendered with color under a given light source, which contains information about texture and shading that helps regularize the optimization of the neural material adaptor.
• Particle binding: Note that NeuMA w/ P.S. does not use the differentiable renderer and, therefore, cannot leverage the particle binding scheme proposed in Section 3.1 of the manuscript. In contrast, our method explicitly models the relationship between 3D Gaussian kernels and simulation particles via the binding mechanism. Since Gaussian kernels generally grow around object surfaces, each kernel encapsulates local object-part information. Our binding mechanism ensures that particles within a kernel share relatively uniform physical states, thereby reducing the degrees of freedom during optimization. NeuMA w/ P.S., however, exhibits a higher degree of freedom as it is directly supervised by the 3D positions of tens of thousands of particles. This increased complexity can make the optimization of NeuMA w/ P.S. challenging, especially given that the trainable parameters are of low rank.

Please note that in our previous experiments, we adopt the same hyperparameter settings and the number of training iterations for both NeuMA and NeuMA w/ P.S. To investigate whether the baseline model is not thoroughly tuned, we conduct additional ablation studies using the BouncyBall benchmark on the learning rate of NeuMA w/ P.S. in the table below. We report the L2-Chamfer distance (L2CD) between the grounded and ground-truth particles here. These values are scaled by $10^4$.

From the table, it is observed that the results obtained by these variants still underperform our method (with a value of 1.19).

It is also worth noting that a previous method, PAC-NeRF [44], also observes a similar phenomenon: A 2D pixel-level supervision better optimizes the physical parameters as opposed to a 3D metric. Based on the above analysis, we argue the experimental results should be plausible.

W4: The lack of real-world examples could be a small but not severe issue to prove the practicability of the proposed method. If possible, the author should provide some realistic examples. If not, the author should claim the reason for the limitation.

Following your suggestions, we supplement dynamics grounding results on real-world data. Specifically, we use the data collected by Spring-Gaus and adopt its experimental setting for experiments. Please kindly refer to Figure A in the attached PDF where we show the grounding results achieved by NeuMA and Spring-Gaus. Note that for real-world data, the ground-truth particles are unavailable and thus we do not present the Chamfer distance here. It is observed that the rendered sequences of NeuMA are more aligned with the observations than those of Spring-Gaus both qualitatively and quantitatively. The results confirm the effectiveness of NeuMA in real-world scenarios.

Thank you for your supportive evaluation of our work. We provide the answers to your question below.

Q1: Does your framework involve the same number of particles if the objects do not deform in normal situations?

Yes. Following the common practice in differentiable physics [25,44,50,88], our method assumes the number of physical particles is invariant across the entire simulation trajectory.

Thanks for identifying our work and providing valuable comments. We try to address your concerns below.

W1: The method appears to not be evaluated on real-world data where assumptions about motion dynamics may not hold perfectly. It would be interesting to visualize the motion residuals and quantify the errors to better understand their characteristics over time. The paper glosses over some relatively minor design decisions such as how is the single view for grounding dynamics chosen, dimensions of the geometry volume and applying the method to a new object.

Real-world data. We supplement the real-world experiment using data captured by Spring-Gaus and present the result in Figure A in the attached PDF. We observe that NeuMA's grounded dynamics are more accurate than the competitor in terms of view synthesis metrics, which verifies the effectiveness of NeuMA in the real world. Note that ground-truth particles are unavailable for real data, so we do not present the Chamfer distance here.

Visualization of motion residuals. We show visual results of motion residuals in Figure D in the attached PDF to quantify the dynamics grounding errors. The color of each particle indicates its sided Chamfer distance to the ground truth. Please also refer to Figure 3 of the manuscript, where we display the average motion residuals per time step over the simulation trajectory. These results verify the superiority of our method over baselines.

Single view Choice. Please refer to Q2 below.

Geometry volume. We use 3D geometry volume, and the grid size is $32^3$.

Generalization procedure. The procedure of applying NeuMA to a new object is: (1) Acquire the 3D Gaussian and surface mesh of the new object from multi-view images (details refer to Section 3.1); (2) Set a user-defined initial velocity and select a material adaptor $Δ\mathcal{M}_\theta$ with the physical prior $\mathcal{M}_0$ to generate the physical-plausible animation.

Q1: Uniformly sampling points inside the object to gather the initial state seems to assume that the geometry is a closed volume. What would happen if the given object is not guaranteed to be closed? Have you considered other sampling methods - for example near the surface? How would these other sampling methods impact quality?

Open volume. Thoughtful viewpoint! We use the Material Point Method (MPM) for differentiable simulation in 3D space and require the 3D volume to be set for each particle. Thus, it is difficult for our method to handle curves or thin surfaces without 3D volume. However, if the object has a well-defined 3D volume, our method should work for either closed or open objects. To verify this, we present the dynamics grounding result of an open container in Figure F in the attached PDF. It is observed our method can address open volume.

Particle sampling. We assume the object is filled with mass, so we use volume sampling instead of surface sampling, since the latter could lead to unrealistic simulations if the interior space is large. For an intuitive understanding, please refer to the results of "NeuMA w/o Bind" shown in Figure 10 of the manuscript. In previous experiments, NeuMA w/o Bind could be considered as using the surface sampling since we directly treat Gaussian kernels (sprinkled near the object surface) as physical particles for dynamics grounding. For objects with thin structures, like the open container in Figure F, surface and volume sampling yield similar results, and thus their dynamic grounding results are comparable. We also present the L2-Chamfer distance (L2CD) between the grounded and ground-truth particles in this case.

Q2: The initial state of Gaussian kernels is constructed from multiple views but the intrinsic dynamics are inferred from a single view. How is this view chosen?

We use multi-view static observations required by vanilla 3D Gaussian Splatting (3DGS) to accurately capture the object's appearance and geometry. For the single-view dynamic observation, we typically select a frontal view of the object and try to minimize self-occlusion. Note that our method also supports multi-view dynamic observation if such data is available.

Q3: How would the technique behave when applied to geometry with uneven mass distribution?

We conduct an experiment on an object with uneven mass shown in Figure E in the attached PDF by assigning particles with different densities, i.e., $\rho$. We also quantify L2CD in this case, and the result is $1.33\times 10^{-4}$. These results show that NeuMA can handle objects with uneven mass.

Q4: What assumptions are made when applying a pretrained NeuMA to a new object?

We assume the new object's 3D representations (i.e., 3D Gaussian kernels and surface mesh) are available. Alternatively, we can do 3D reconstruction to get these representations given dense multi-view images. The initial velocity is also required but is user-defined.

Q5: What are some failure cases where the initial state estimate may not produce good results?

When acquiring the 3D representations, our framework inherits the common failure cases from multi-view reconstruction techniques, e.g., (1) the camera parameters are inaccurate, and (2) the static captures are too sparse. Besides, the prediction of initial velocity may be inaccurate if the object exhibits complex motions at the beginning (e.g., different parts have different initial velocities), as we currently assume a uniform initial velocity.

Limitation: Large objects and non-convex objects.

Nice suggestion! In Figure G, H in the attached PDF, we show the dynamics grounding results of a large object and a non-convex object separately. The quantitative results are $0.91\times 10^{-4}$ and $0.51 \times 10^{-4}$. The results verify the effectiveness of NeuMA on complex geometries.

We are grateful for your positive and constructive comments, and try to address your concerns below.

W1: It is a bit unclear to me how the generalization to novel objects and object interaction (in section 4.4) should be evaluated ... Can these experiments be quantified and compared to other methods?

We supplement quantitative results on dynamics generalization by evaluating the L2-Chamfer distance (L2CD) between grounded and ground-truth particles in the simulation space. Note that we scale L2CD by $10^4$ in all experiments (same below).

In this table,

• Bouncy -> "N" means that we apply the learned NeuMA on BouncyBall directly to the letter "N" and evaluate the generalization performance (the same with Rubber -> "N" and Sand -> "N").
• Ball & Cat is the quantitative result of the interaction between the ball and the cat in Figure 7 in the manuscript.

From the table, we observe that NeuMA achieves favorable performance over other methods in dynamics generalization.

W2: It is unclear what role $α$ plays in the NeuMA approach. From the provided examples in Figures 8 and 9, it seems that (qualitatively), reconstruction improves as more weight is given to $Δ\mathcal{M}_𝜃$. This raises the question, what is the benefit of $\mathcal{M}_0$? Have the authors done experiments with only the learned model?

We implement the neural material adaptor $Δ\mathcal{M}_\theta$ using the low-rank adaptation (LoRA) and $r,α$ are two hyperparameters for LoRA. Specifically, $r$ is the rank of the trainable weights and $α$, which is commonly set to equal to $r$ during training, can be tuned during inference as a scaling parameter to modify the influence of LoRA on the base model. In our case, $α$ is like a weight coefficient on the adaptor: When $α=0$, it means we do not alter our prior (i.e., the base model $\mathcal{M}_0$); when $α$ gets larger, the generated dynamics will become more similar to the given observation. In general, LoRA could not be evaluated alone without the base model. Please refer to Q2 below, where we study the benefit of $\mathcal{M}_0$ by changing different $\mathcal{M}_0$ for dynamics grounding.

Q1: Emphasis is placed throughout the paper on single-camera inputs, but the initial shape creation requires multiple views. Is this not contradictory?

We emphasize the use of a single camera as this eases the burden of camera synchronization for capturing the full dynamic trajectory, which is required by previous works like PAC-NeRF. During the initial state acquisition, we can use a single camera to capture multi-view images to ensure an accurate modeling of the object appearance and geometry for later visual grounding. Moreover, thanks to the physical prior $\mathcal{M}_0$, we have a rough guess of the object motion and could perform dynamics grounding given single-view camera observation. We will clarify our setting in the revision.

Q2: Does the material need to be known for correct application of $\mathcal{M}_0$? What is the effect of a misalignment of actual material and the heuristically selected model?

In previous experiments, we assume a correct material model is known as the prior $\mathcal{M}_0$ for dynamics grounding. It should be noted that the underlying material parameters (e.g., Young’s modulus and Poisson’s ratio) are unknown. Here, we study the effect of inaccurate application of $\mathcal{M}_0$. We choose two plastic objects, RubberPawn and ClayCat, for this experiment. The specific settings and L2CD results are shown below. We also show the visual grounding results for RubberPawn in Figure B in the attached PDF.

In this table,

• Setting I refers to our previous experimental setting in which the correct material model is set as the physical prior.
• Settings II and III adopt inaccurate elastic material models, but the resulting motion still conforming plastic material.
• Settings IV and V are more challenging, as the former is commonly used for simulating elastic objects and the latter for granular objects.

From the table, we can see that our method can handle moderate deviation from correct material models (e.g., Setting II and III). However, when the physical prior is completely wrong, (i.e., Setting IV and V), the performance would undergo an obvious decrease. As a remedy, it may be helpful to leverage Large Language Vision Models (LLVM) like GPT-4o to give a plausible guess of the material models given some key frames of the visual observation.

Q3: Does the binding between simulated and GS particles need to be updated as timesteps increase and the material deforms substantially?

For efficiency, we only compute the binding matrix during the initial stage. This strategy also works well for materials with large deformations (e.g., SandFish from our synthetic data).