Learning rigid-body simulators over implicit shapes for large-scale scenes and vision

We thank the reviewer for the helpful feedback and insightful comments.

Representation of methods like MeshGraphNet and DPI is misleading. SDF-Sim cannot resolve deformable solids and fluids

We fully agree that MeshGraphNet and DPI can handle broader types of physical systems. We are happy to add a more balanced statement to reflect that.

We would like to clarify that when we draw comparisons mesh-based GNN simulators, we generally refer to FIGNet and FIGNet* as these two models have shown the most complex rigid dynamics so far among the learned models. MeshGraphNet and DPI have shown amazing results in the realm of deformable simulation, however rigid bodies have their own unique challenges. Rigid collisions and contacts are non-smooth and are notoriously hard to derive numerical approximations for. Rigid dynamics is chaotic in nature: tiny errors in the model or in the initial states can quickly accumulate and lead to large deviations in later steps of the object trajectory. Hence, in this work we focus on rigid bodies only.

A clarification about the construction of the collision edges.

The reviewer’s understanding is entirely correct: collision edges are created dynamically at every step of the simulation.

Constructing SDFs: Did the paper construct a narrow-band SDF or the full SDF on the whole grid as the training data for the network to fit?

We don’t use SDF grids for training, as the reviewer might be suggesting. Instead, we randomly sample the ‘query’ points around the object by taking the points on the object surface and adding gaussian noise with zero-mean and sigma=0.1. We compute the SDFs for these query points and use them for training learned SDFs. Therefore, learned SDFs are not constrained to be narrow-band, although most of the training points fall within $2 \sigma = 0.2$ distance from the object surface. Figure S1(d-e) shows that learned SDFs are most accurate within [-0.2, 0.2] distance interval from the surface. In practice, we only care about the accurate SDF estimates within the collision radius=0.1, therefore the current SDF accuracy is sufficient.

We provide more details on SDF training in Appendix D.1.

Did the network encourage unit SDF gradient norm during training?

We did not find that it is necessary to specifically encourage the unit SDF gradient norm. We see this property naturally emerging during the training. We think it is due to our training strategy where we randomly sample the query points near the object surface instead of training on grid points.

Computing closest points: Equation 1 is correct in theory. In practice, … one needs to apply Eqn 1 multiple times before obtaining a fairly accurate closest point.

Empirically, we find that it is sufficient to apply Eq. 1 once to find the closest point on the object surface, thanks to the accurate learned SDFs. We use only one projection step in all our experiments. We invite the reviewer to inspect the Figure S1(a), where we compare the true closest point to the one-step projection* from a learned SDF and show that the mean-squared error stays within [1e-4, 1e-3] for different network sizes.

O(K^2) complexity of SDF-based inter-object edges: … simulators rarely go over all O(K^2) pairs in a brute-force manner.

Here we refer to the number of the constructed edges between the objects. Within the collision region, FIGNet an FIGNet* connect all mesh triangles of one object to all mesh triangles on another object, so the number of edges between the objects is O(K^2). In SDF-Sim we connect nodes of one object to the center of another object (within the collision region), so the number of edges grows only linearly in the number of nodes O(K).

For mesh-based simulators, K^2 edges would be a worst-case scenario in case the collision region is huge and spans the entire scene, which is indeed unlikely to happen.

However, we can see that these asymptotic trends clearly manifest in Figure 7. The quadratic number of edges in FIGNet pose a significant problem, leading to 1M collision edges for 160 objects and causing the model to run OOM. SDF-Sim has <100k edges for the same simulation.

Spheres-in-Bowl and Shoes use repeated objects. These scenes … did not represent the “true” large-scale rigid-body simulation with …many rigid bodies with various shapes.

We have an example of a large-scale scene with many objects with various shapes. It is shown in Figure 10 and replicated in the attached PDF for reviewer’s reference. The corresponding simulation video is shown on the project webpage (“Heaps of stuff” section). This simulation includes concave shapes (shoes, hanger) and thin structures (screwdriver, baking form). We will highlight it more in the final manuscript.

Spheres-in-Bowl: Although sphere SDFs are trivial, Spheres-in-Bowl is a nice example to study the scaling properties of the models due to a large number of collisions within any given time step. We show other large-scale examples with more complex shapes, such as a pile of shoes (Figure 2), a pile of knots (Figure 10 top) and a mix of different objects (Figure 10 bottom). Finally, the bowl itself does not have an analytical SDF expression and still requires an SDF for collision detection.

Lines 69-71 suggest acceleration structures like BVH often rely on CPU implementations that are difficult to accelerate. BVH trees on GPUs are actually common.

Thank you for pointing this out – we will remove this claim from the paper.

We thank the reviewer for the constructive feedback. We will add the discussions below into the paper. We will also correct the reference to FIGNet* and DANO description in the related work.

Are euler integration and shape matching differentiable?

Both Euler integration and shape-matching (using differentiable SVD) are differentiable, and we could potentially include those steps into the learning process. However, we suspect that the current loss on the nodes before shape-matching provides a fine-grained learning signal and forces the model to correct errors for each node individually.

Edge pruning in FIGNet$\*$ is fairly adjusted? Influence on Fig 7? Can FIGNet$\*$ outperform SDF-Sim overall?

The edge pruning has one main parameter, which is the collision radius. We use the same collision radius of 0.1 for FIGNet, FIGNet* and SDF-Sim, matching the collision radius used in FIGNet and FIGNet* papers.

If we change the collision radius, we expect the relation between the models in Figure 7 to stay the same, because FIGNet* constructs O(K^2) edges within the collision region, while SDF-Sim has only a linear number of edges O(K), where K is the number of nodes. Given such asymptotic complexity, we do not think FIGNet* can outperform SDF-Sim.

Is the simulator real-time capable?

We perform the simulation for 200 time steps, 5 seconds of total simulation time.

We have added a runtime comparison to PyBullet in the attached PDF (Figure 2). SDF-Sim has a similar runtime to PyBullet up to ~30 objects per scene. Generally, learned simulators are not real-time, except on very small scenes.

We note that optimizing our code for speed was not our goal. MuJoCo or Bullet use low-dimensional collision meshes to achieve real-time, while all of our large simulation scenes use relatively detailed node sets. Additionally, a large part of SDF-Sim runtime is spent on constructing the edges of the input graph, which can be sped up by faster edge pruning, batching SDF queries or using specialized cuda kernels.

Comparison to the analytical simulators such as NVidia PhysX, MuJoCo, bullet

This is an interesting question. Note that we use Bullet simulations as the ‘ground-truth’ in experiments with Movi B/C (Figure 6) and Spheres-in-Bowl (Figure 8). As shown in Figure 6, SDF-Sim remains consistent with the Bullet simulation, having low translation and rotation error after 50 rollout steps.

In the attached PDF, we also added new comparisons to Bullet in terms of runtime, penetrations and simulation stability (Figures 2 and 3).

We note that it was not our goal to outcompete the state-of-the-art analytical simulators in runtime. We develop learned simulators because they have their own unique advantages that analytical simulators don’t provide, specifically for Robotics. The main difference is that learned simulators can be trained directly on real-world observations. They can track the real object trajectory better than analytical simulators, solving a well-known sim-to-real gap [1]. Another common issue is precisely estimating the initial states, which analytical simulators rely on – learned simulators can compensate for these inaccuracies [1]. Finally, learned simulators are differentiable and can be used for optimization and design [2]. At the same time, we agree that learned simulators were not optimized for runtime and are slower than analytical simulators. Therefore, in this paper we chose to only compare to other learned simulators.

[1] Graph network simulators can learn discontinuous, rigid contact dynamics. Allen et al. CoRL 2023

[2] Inverse Design for Fluid-Structure Interactions using Graph Network Simulators. Allen et al. NeurIPS 2022.

Sec. 4.4. comparison to PhysX, MuJoCo, bullet on SDFs from vision, where extracted SDF is transformed into a mesh

We expect that PhysX, MuJoCo, Bullet to be able to successfully handle the scene, since 80k nodes is feasible for these simulators.

We are not claiming that SDF-Sim is the only simulator that could do that. Instead, this experiment shows that 1) SDF-Sim can generalize to new scenes, despite being trained on synthetic data 2) We can plug in output from VolSDF directly into SDF-Sim, bypassing the mesh conversion 3) FIGNet already runs OOM on this scene

We will remove the wording “large” for this section and clarify the goals of this experiment in the paper.

The requirement of watertight objects can be a disadvantage too, if the objects are thin

Modeling thin surfaces, e.g. cloths, is non-trivial, and it is outside of scope for this paper. Here we focus on modeling rigid objects, for which volumetric shapes are often a good approximation.

Although SDF-Sim is not designed to represent cloths, other graph-network simulators, e.g. MeshGraphNet [1] are able to do so. One can imagine combining the ideas from SDF-Sim and MeshGraphNet, where rigid objects are represented as an SDF for efficiency, while thin/deformable objects are represented as a meshes.

[1] Learning mesh-based simulation with graph networks. Pfaff et al. ICLR, 2021

A simple MarchingCubes reconstruction might be sufficient to extract a mesh from VolSDF

Extracting meshes intended for simulation require special handling for two reasons:

1. Naive extraction with Marching Cubes leads to noisy meshes. VolSDF is a learned model and its zero values might not be exactly on the surface. Therefore, the meshes produced by MarchingCubes can contain holes or floating faces and might require substantial clean-up before they can be used in the simulator, because analytical simulators rely on the meshes to be watertight to perform inside-outside tests.

2. Analytical simulations need a “collision” mesh consisting of convex subcomponents, requiring an additional convex mesh decomposition which is often done manually.

These steps are feasible to do, but they require knowledge and tools that a deep learning researcher might not have encountered before.

The author response has addressed my concerns. I have no further questions at this point.

We thank the reviewer for positive evaluation of our paper and for insightful comments.

Learned SDFs …. accuracy is limited by the network's capacity.

We agree that the accuracy of Learned SDFs depends on network capacity. However, we found that even a basic MLP with 8 layers and 32 units per layer is sufficient to achieve <10e-5 error on SDF estimates and perform accurate simulation (Figures 9(a-c)). We did not find the SDF network capacity to be a limitation.

Every object's SDF-model need to be pushed on the GPU--or swapped in-and-out

Evaluating each SDF sequentially and swap them in-and-out would be very inefficient and slow.

SDF weights are small, and we keep all object SDF on the GPU throughout the simulation. Even with many different object shapes in a given scene, keeping SDFs in GPU memory is not an issue. To compute the distances, we also evaluate all SDFs in parallel, leveraging the GPU accelerators. As mentioned in Limitations, it is possible to further reduce the memory consumption, if needed, by using amortized models, weight sharing, or faster SDF queries.

This work seems to use mostly a smaller set of repeatable object

We emphasize that SDF-Sim is not specific to using repeatable objects. We provide an example of a large-scale simulation with many different objects in Figure 10 in the original paper (also replicated in the Rebuttal PDF for reviewer’s reference).

Why is it important to build learned simulators in the context of rigid-object simulations? Are the simulators also memory-constrained?

This is a great question. The limitation of traditional simulators like MuJoCo or Bullet is that the simulations inevitably diverge from observations of real objects – a so-called sim-to-real gap [1]. The traditional simulators rely on hard-coded approximations of physical interactions that might not match the properties of real objects. It was shown that even with careful parameter tuning, the analytical simulators cannot precisely model a real cube tossed on a table [2]. This is a major problem for robotics, which heavily relies on simulated environments, and a long line of research in robotics is dedicated to mitigate the sim-to-real gap.

In contrast, learned simulators do not suffer from sim-to-real gap, as they can be directly trained or fine-tuned on real-world data. In fact, learned simulators can be better at tracking real objects than traditional simulators like MuJoCo, Drake or Bullet, even in the low data regime [3].

Another advantage of learned simulators is that they are differentiable by nature and can be used for solving inverse problems, such as design or control (as noted in reviewer’s question below).

However, learned simulators used to be memory-constrained and work only on small scenes with up to 10 objects – this is exactly the problem we are targeting in the paper.

A learnable surrogate model … can also be useful as a differentiable simulation, e.g., in order to optimize the scene or some objects.

This is a great point. SDF-Sim is fully differentiable: GNN and learned SDFs are differentiable by nature of neural networks. We can also pass gradients through the construction of input graph from GNN to the learned SDFs. One can use a differentiable version of SVD for the shape-matching step.

Generally, GNN-based models can be successfully used as differentiable simulators to optimize an object shape, such as an airplane wing, and can provide more stable gradients than traditional differentiable simulators [4]. We believe that SDF-Sim inherits these properties and can be used for differentiable simulation in the areas of mechanical design, robotics and more.

The vision experiment, simulating in an environment extracted from a multi-view NeRF reconstruction, is a bit underwhelming. I am unsure what is the main point of this result.

This experiment is a proof-of-concept that demonstrates 1) SDF-Sim can generalize to real-world object shapes derived from 3D reconstruction, although it was trained on “clean” shapes 2) we can directly connect the VolSDF outputs to SDF-Sim, making a fully-differentiable vision-to-simulation pipeline.

The alternative approach to simulating this scene would be to convert VolSDF output into a mesh and run a mesh-based simulator. However, generating clean meshes can be tricky and the meshes may need to be cleaned-up before they can be used in the simulator. Additionally, most rigid solvers like Bullet or MuJoCo rely on convex meshes, requiring additional convex mesh decomposition which is often done manually. In contrast, SDF-Sim directly takes SDF as an input, and we can bypass the step of converting SDFs into meshes.

Would the proposed method also work for open surface objects, using their UDFs (unsigned distance functions)? Can it detect penetration?

We hypothesize that SDF-Sim will work with UDF representations, assuming that UDFs are used for all objects and SDF-Sim is trained on UDF representations.

The learned simulators generally are able to detect penetrations using unsigned distances. For instance, mesh-based FIGNet baseline uses only the unsigned distances between individual triangles and can successfully reason about penetrations by pooling the information from pairs of faces on the surfaces of the two objects. In the current SDF-SIm paper, we performed an experiment with unsigned distances computed from a mesh (Appendix E.6) and found that the accuracy is similar to our current model with SDF representations.

[1] What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery. Huang et al.CoRL 2023

[2] B. Acosta, W. Yang, and M. Posa. Validating robotics simulators on real-world impacts. IEEE Robotics and Automation Letters, 2022.

[3] Graph network simulators can learn discontinuous, rigid contact dynamics. Allen et al. CoRL 2023

[4] Inverse Design for Fluid-Structure Interactions using Graph Network Simulators. Allen et al. NeurIPS 2022.

We thank the reviewer for highly appreciating the value of the paper and for valuable comments.

The wording "small-scale datasets"

Indeed, here we mean the datasets have small-scale scenes with up to 10 objects, while later in the paper we scale the simulator to scenes with hundreds of objects. We will update the wording in the paper.

What is the difference between smaller scenes and larger scenes?

The key difference is that having an order of magnitude more objects in large scenes leads to many more collisions. We specifically design large scenes to have stacked objects, as this setup is considered to be a particularly hard problem in rigid simulation. It requires resolving many collisions within the same time step and propagating the collision response across a chain of multiple objects. In contrast, in small scenes, the collisions are sparse and involve only pairs of objects.

We suspect that FIGNet* and FIGNet may have some slight accuracy advantage in pairwise collisions due to the explicit computation over the mesh. SDF-Sim is better at propagating those collisions across many objects and therefore has better performance on more challenging scenes with stacked objects.

Appendix E.6: What do you mean by "train an SDF-Sim architecture using the accurate distances directly computed from a mesh"? Why do an experiment with unsigned signed distance?

Thank you for pointing out the confusion – we will update the section to make it clearer.

The distances predicted by learned SDFs may not be perfectly accurate, because they come from a learned model. In Appendix E.6 we aim to test whether this is an issue for the simulator. To do so, we compute the distances using a brute-force distance computation between query points and each triangle in the mesh. Then we train the simulator using these distances instead of learned SDFs. It is tricky to estimate the SDF sign using this approach (whether the point is inside/outside of the mesh), so we use the unsigned distances. We found that using learned SDFs versus pre-computed distances makes little difference in practice.

Surface nodes … are the mesh vertices used for collision simulation? In Appendix E.5 surface nodes are samples from the SDF. Why isn't it the default design? Is there a trade-off?

This is correct – for most experiments in the paper we used mesh vertices from the original collision meshes as surface nodes for SDF-Sim. The only exception is the experiment in section E.5, where the nodes were sampled from an SDF.

Indeed, we have shown that we can potentially scale the simulator further by re-sampling the nodes from an SDF. However, the tradeoff of node re-sampling is that it introduces an additional step of the pipeline with its own set of hyperparameters (grid size, downsampling, etc.) that might need to be tuned for more complex shapes. In the main paper we chose to focus on a clear message that we can scale the simulator solely by using SDFs and adapting the GNN, as this architecture works universally well for different object shapes.