GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation

We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.

---

1. Some symbols are not defined clearly, making it hard to follow at times. For example, $Discretize$ operator in Line 214; $\tilde{P}$ and $F$ are not defined in the text. I need to guess from Alg 1.

Reply: Sorry for the lack of clarity. $Discretize$ denotes the operation mapping particle positions to voxel indices on the density field. \tilde{P}$and$F$ are the sampled particles and the density field. We add a table to the attached PDF in the "Author Rebuttal" session to clarify the operators and symbols in detail. We will also add the table to the Appendix in the revised version.

---

2. If the 4D geometry is accurate enough, is it adequate to only use 3D supervision? Ablation studies are needed to validate the necessity of 2D mask supervision.

Reply: Thank you for this constructive advice.

(1) To answer the first question, we perform system identification on the torus object, which is the only instance that provides a ground truth mesh model in the PAC-NeRF dataset. Specifically, we use the ground truth point clouds as the continuum for simulation and utilize the mesh model to extract surface particles from the point cloud as 3D supervision. The experiment, performed with this configuration and other settings unchanged, achieves $E = 1,002,907.25$ and $\mu = 0.2991$, which are close to the ground truth ($E = 1,000,000$ and $\mu = 0.3$). Therefore, we believe that it should be sufficient to use 3D object surfaces as supervision once the recovered geometry is accurate enough.

(2) To evaluate the necessity of 2D mask supervision, we perform system identification on 45 cross-shaped object instances in the PAC-NeRF dataset by our method but with only object surface supervision. The results are reported in the table below. It is obvious to see that combining both 2D and 3D shapes as supervision can achieve more accurate performance compared to using 3D shapes only. Therefore, we believe that utilizing 2D mask supervision to some extent makes up for the errors introduced by the 3D object surfaces extracted from dynamic 3D Gaussians. We will add the analysis to the revised version.

---

3. Coarse-to-fine density field creation: At the beginning, the reconstruction contour is much larger than the object. How does it shrink to the actual boundary? The $TrilinearInterpolation$ operator will not shrink the contour, and there is no operator to assign zeros.

Reply: Sorry for the lack of clarity. Although all the operations cannot assign voxels outside an object to zeros, the trilinear interpolation and mean filter operations can reduce the density of the voxels outside the object boundary. By iteratively performing mean filtering and particle voxel reassigning, the densities outside the boundary will be sufficiently small while the object boundary and internal region keep high-density values, and we thus can extract the object particles by thresholding the density field.

---

4. In the simulation, how do the Gaussian kernel scales evolve? The paper seems to assume isotropic kernels, but physical deformation can transform the sphere into an ellipse. How is this addressed?

Reply: Sorry for the lack of clarity. In this work, we use the grid size of the density field as scale attributes of Gaussian kernels and fix them during the simulation. We admit that a physics-informed scale transformation such as PhysGaussian [1] allows a more realistic rendering. In future work, we will integrate this function into our method to enable kernel transformation during simulation.

---

[1] Xie, Tianyi, et al. "Physgaussian: Physics-integrated 3d gaussians for generative dynamics." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.

---

1. The authors stated that such a lightweight architecture of the motion-factorized dynamic 3D Gaussian network is sufficient for complex motions rather than modeling each basis with an independent network (line 168-171) while lacking an experimental proof.

Reply: Thank you for pointing out this question. We conducted two ablation analyses to empirically demonstrate the arguments mentioned in lines 168-171.

(1) We first use our method to perform dynamic Gaussian reconstructions on 45 cross-shaped objects in the PAC-NeRF dataset, except each motion basis is modeled with an independent network. Specifically, each basis takes the encoded time as input and contains 8 fully connected layers. The output is the residuals of position and scale of this basis. We use the setting mentioned in Sec. 5.1 to evaluate the CD and EMD on the above method variants and compare them with our method. The results are reported in the table below. The results show that our backbone-shared architecture (Ours) slightly outperforms the independent-basis networks (Ind.) in terms of dynamic reconstruction, which empirically demonstrates the strength that the reduced order modeling of dynamic Gaussians is sufficient for motion reconstruction tasks. Moreover, our method can achieve less training time compared with the independent design (around 15 minutes vs. 45 minutes for a dynamic scene on a single 3090 GPU).

(2) To validate the second argument in lines 171-172, we compared our method with DynMF [1], which also uses neural networks as learnable bases while considering motion coefficients as time-invariant Gaussian attributes, by evaluating the PSNR on the D-NeRF dataset. The results are reported in the table below. The results show that modeling the motion coefficients as time-variant variables does increase the ability to fit the dynamic scenes.

---

2. The representation in Section 4.3 is lacking in elaboration and should accompany with more details in the supplementary. Please give some detailed elaboration for Section 4.3 about Gaussian-informed continuum, e.g., notations used in Algorithm 1, dimensions of the variables, etc.

Reply: Thank you for this constructive advice. As suggested, we added a table to the attached PDF in the "Author Rebuttal" session to clarify the operators and symbols in detail. We will also add the table to the Appendix in the revised version.

---

3. It would be nice to see a comparison between the choice of Motion-factorized dynamic 3D Gaussian network and previous architectures ?

Reply: Please refer to the first reply.

---

4. It is important that the experimentation on complex motions must be performed in order to truly understand the strength of the proposed method.

Reply: Thank you for this suggestion. We conduct an additional experiment on a scenario with a more complex boundary condition and motion trajectory. Specifically, we use our method to perform system identification on an elastic rope falling onto two rigid cylinders. The data format is the same as PAC-NeRF. The estimated property is reported in the table below, and the simulated trajectory is visualized in the attached PDF. The results show that our method can also generalize to scenarios with more complex boundary conditions and motions.

---

[1] Kratimenos, Agelos, Jiahui Lei, and Kostas Daniilidis. "Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting." arXiv preprint arXiv:2312.00112 (2023).

We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.

---

1. In the real-world experiment, I found the authors switched to FEM for deformable body simulation, which conflicts with the MPM simulator used in their pipeline. I think it needs justifications.

Reply: Sorry for the confusion caused. Switching to FEM is because most of the widely-used robotic simulators incorporate FEM for deformable object simulation, including Isaac Gym, the simulator we used in our experiments. Moreover, estimating physical parameters using MPM and then applying them to FEM should be a feasible way for this application, since

(1) both the Material Point Method (MPM) and the Finite Element Method (FEM) originate from Galerkin methods [3]. In theory, MLS-MPM can achieve first-order consistency in simulations [3], which is comparable to the consistency achieved by linear FEM simulations,

(2) the material properties (e.g., Young's modulus and Poisson's ratio) are independent of the numerical methods.

We will clarify this in the revised version.

---

2. Some wordings are confusing: e.g., line 166, do you mean effective instead of efficient?

Reply: Sorry for the confusion caused. Indeed, we should use "effective" in line 166. We will correct the expression in the revised version.

---

3. I wonder what makes the difference between the proposed method and PAC-NeRF in the infilled particle generation.

Reply: Sorry for the lack of clarity.

(1) PAC-NeRF samples particles by directly selecting NeRF field voxels whose alpha values are greater than the predefined threshold and then performing uniform sampling 4 times on each extracted voxel. Since PAC-NeRF defined the alpha threshold with a small value to make sure that a solid continuum can be obtained, they usually tend to recover over-large shapes.

(2) On the contrary, our method turns to generate a density field based on the Gaussian and initial particles {$\mu(t)$} $\cup P_{in}$. Among these particles, the Gaussian particles are prone to be located at the object surface, which can guarantee high density at the surface region, while the initial ones incorporated with the coarse-to-fine operations can ensure a solid internal region. With this module, we can extract the continuum from the density field, which serves only for representing the object shape instead of the NeRF field, which is utilized for rendering. Please refer to Figure 7 in Appendix A.2.2 to see the qualitative results of the proposed filling algorithm, along with ones from PAC-NeRF.

We will add more explanation in the revised version.

---

4. If I understood correctly, the motion network only takes time as input. I wonder if it helps by combining the temporal encoding from DiffAqua [1] to further capture the low and high frequencies.

Reply: Sorry for the lack of clarity. We do employ temporal and positional encoding to the time $t$ and position $\mu_0$, respectively, to introduce features with various frequencies. Specifically, the encoding module is denoted as $\gamma(x) = \left( \sin(2^k \pi x), \cos(2^k \pi x) \right)_{k=0}^{L-1}$, where $L=10$ for both $t$ and $\mu_0$, which is exactly the same as the setting in DiffAqua. We will add the notations to Figure 2 and the implementation details to Appendix A.1.1 in the revised version.

---

5. The infilling algorithm seems expensive since the complexity grows exponentially. Did you try using Octree [2] or similar algorithm to speed it up?

Reply: Although the memory requirements for processing the volumetric data scales cubically with the grid resolution, we still use the naive volumetric data structure for our algorithm because

(1) in practice, only four iterations are required to achieve sufficient accuracy (the implementation details are also available in Appendix A.2.1),

(2) such a data structure can be efficiently implemented with GPU acceleration based on PyTorch, where the trilinear interpolation and mean filter operations are implemented by "grid_sample" and "conv3d" functions, respectively.

Therefore, we can almost achieve real-time performance (more than 10 fps on a single Nvidia 3090 GPU) on our infilling algorithm.

---

[3] Hu, Yuanming, et al. "A moving least squares material point method with displacement discontinuity and two-way rigid body coupling." ACM Transactions on Graphics (TOG) 37.4 (2018): 1-14.

We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there’s anything we can clarify further.

---

0. Some of the technical terms are misused, making the paper confusing to read. For example, "implicit shape representation" is repeatedly used to refer to the rendered object image masks, but "implicit" generally refers to using functions (parametric or neural networks) to represent shapes, where shapes must be retrieved through function evaluations, hence "implicit". …

Reply: Sorry for the confusion caused. Since our method employs both 3D surface particles and 2D object masks for supervision, we originally intended to use "implicit shape representation" to describe 2D object masks in order to distinguish them from 3D surface particles, but we overlooked the ambiguity it introduced. We will correct the description by directly using 2D object masks in the revised version.

---

1. Table 3: Can you include the ground truth values in the table so that readers understand what to expect?

Reply: Thanks for this constructive advice. For more details about the estimated and ground truth values, please refer to the attached PDF in the "Author Rebuttal" session. We will also add the table to the Appendix in the revised version.

---

2. System Identification: How do you set the initial parameters for the system identification? Are the optimizations sensitive to initial conditions?

Reply: The table in the attached PDF also lists each instance's initial guess. To make a fair comparison, we followed the setting in PAC-NeRF [1] to assign the same initial values for the instances with the same material. The results show that our method is robust to initial conditions even when they are significantly different from the ground truth.

---

3. Figure 4: Would mask-based supervision fail when the estimated shapes are significantly different from the ground truth? Have you observed any cases where this occurs?

Reply: Under extreme conditions, the mask-based supervision might fail when the simulated trajectory is completely out of view for all viewpoints. However, we did not find any failure cases in our experiments, because we performed initial velocity estimation before system identification. With the initial velocity available and multiple viewpoints located at proper positions, we observed that it's unlikely that the simulated trajectory is outside the field of view, and the estimated shapes will always converge to the ground truth shapes even if they have significant discrepancies at the initial stage.

---

4. Figure 1a: There is a typo in the figure caption. Change "caption" to "capture.”

Reply: Thank you for pointing this out. We will correct the typo in the revised version.

---

5. References: It would be beneficial to include references to works on differentiable cloth simulation for system identification and inverse problems, such as "Differentiable Cloth Simulation for Inverse Problems" by Liang et al. and "DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact" by Li et al.

Reply: Thank you for this suggestion. These two works also tackle the inverse problems with differentiable simulators, particularly in cloth material. We will cite the related works you mentioned in our revised manuscript.

---

6. Limitations: I do wonder about the failure cases of the method if there are any.

Reply: In the synthetic experiments, we didn't encounter any failure cases on both the PAC-NeRF [1] and SpringGaus [2] synthetic datasets, since each scenario includes video sequences from 10 or 11 distinct viewpoints, which are sufficient for dynamic Gaussian reconstruction. However, the proposed dynamic Gaussian module would fail to reconstruct the trajectory on the SpringGaus real dataset because it only contains 3 viewpoints to capture the dynamic scene. That's why we only use object masks for the supervision of system identification (More details are illustrated in Section 5.3 and Appendix A.5.). Therefore, our method might not perform well when fewer views of the scenario are available. Using fewer views to recover both geometry and system identification is more practical and will be an interesting direction for future work.

---

[1] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen-fanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.

[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. arXiv preprint arXiv:2403.09434, 2024.