Latent Task-Specific Graph Network Simulators

We thank the reviewer for their constructive feedback and thoughtful suggestions. This rebuttal addresses the inclusion of recent advancements in the related work and as a baseline comparison, and clarifies design choices such as the use of collider states and context information. Additional details on architectural components and trajectory prediction mechanics are also provided to enhance clarity and completeness.

Responses to Specific Concerns

1. This paper is highly related to the Graph-based Neural Simulators. However, in the related work section, the latest advancements in this field are not included, and most of the work discussed is from 2023 or earlier. This could make the paper appear somewhat outdated. I believe this section could benefit from a more comprehensive overview of the field, especially more works from 2024. Below are two of the latest advancements about Graph Network Simulators that I recommend the authors to discuss them in Section 2.1 ,or better, use them as baselines for comparison. However, given the tight rebuttal timeline, it is also tolerated that concurrent works were not included for comparison. (1) "DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering" 2024 .. This work integrate traditional Newton mechanics into the graph network design to benefit from mechanics priors for longer term prediction. (2) "Equivariant graph neural operator for modeling 3d dynamics" 2024 .. This paper deal with dynamic prediction tasks as trajectory-level rather than next-step level by operator learning, which is somewhat relavent with this reviewing work. Also, it handle the equivariant issues.

We thank the reviewer for highlighting the importance of including recent advancements in Graph Neural Simulators. The related work section has been updated to discuss both DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering and Equivariant Graph Neural Operator for Modeling 3D Dynamics. EGNO has also been implemented as a baseline, but due to the short rebuttal timeline, its evaluation is not fully complete and will be included in the final revision on Wednesday. Additionally, we discuss AURORA, a foundation model approach for climate predictions, as an alternative to meta-learning.

2. For Equation 3, does it use past trajectory collider states when encoding z because I saw that you seem to only use the latest state, or does it rely solely on the historical information of the deformed object? I believe it would be more reasonable to use all the historical information of the collider here as well, since the deformation of the mesh is passive.

We thank the reviewer for pointing out this potential ambiguity. The model does incorporate the historical trajectory of the collider when encoding z, ensuring that both the past collider states and the deformation history of the object are considered. The paper has been updated to clarify this aspect.

3. If this method is trained on an elastic dataset, can it generalize directly to elastoplastic materials? I believe it would be worthwhile to discuss the generalization across different materials in the experiments, rather than limiting it to variations in mechanical parameters within the same material.

The proposed method has not been tested on elastoplastic materials, as this setup was not included in our current data suite. However, we consider this an interesting direction for future research. If the tasks share common aspects and the context sufficiently captures the relevant properties of the new material, the model should, in principle, be able to generalize across different material types. This hypothesis aligns with the adaptability demonstrated by the method in other scenarios: to evaluate the model's generalization capabilities, we created a new data split in the Planar Bending task. Here, the Young's Modulus values used for training ranged between 60 and 500, while the test set included Young's Modulus values from [10, 30, 750, 1000], representing a clearly out-of-distribution scenario. Results from these experiments, included in the appendix, show that M3GN significantly outperforms both MGN and MGN (Oracle) in this setting, demonstrating strong generalization to out-of-distribution material properties. The reviewer can find the results in Figure 5.

We thank the reviewer for their thoughtful feedback and constructive comments. The points raised have helped us identify areas for clarification and improvement, which we have addressed in the revised manuscript. In this rebuttal, we provide detailed responses to the specific concerns, including clarifications on the physical proximity used for edge creation, the basis function representation, training/test splits, and further elaboration on runtime comparisons. We hope the revisions will provide a clearer understanding of the methodology and enhance the overall quality of the paper.

Responses to Specific Concerns

Some descriptions are unclear and some important details are missing. (1) in line 242, "graph edges between the deformable object and collider are added based on physical proximity to model interactions between objects." what is the physical proximity exactly? Since the deformation mesh node position for the end timestep is unknown, I suppose we cannot use that to compute the distance. Whether this edge creation is done only for known timesteps or if it's updated during prediction?

We thank the reviewer for highlighting this point. The "physical proximity" refers to the creation of an edge between the deformable object and collider when the distance between the two is smaller than a given threshold of 0.3. We updated the hyperparameter section in Appendix C to include this value for clarity. This edge creation process is applied only to the context. During prediction, we rely on the anchor step and the previous context to predict the entire remaining trajectory. Therefore, no information about the proximity of future steps is required when unrolling the ProDMP trajectories.

(2) in line 231, why is the term c_1y_1(t) + c_2y_2(t) only depending on the inital conditions? What is the representation of the pre-computed basis fuction \phi?

We briefly reply to this question in below, while added a detailed presentation of the ProDMP theory in Appendix A.

ProDMP, as a parameterized trajectory generator, models a trajectory using a second-order dynamical system. This system is governed by a second-order linear ordinary differential equation (ODE). ProDMP builds upon its predecessor, DMP, which computes the trajectory by applying numerical integration from the start to the end of the trajectory. In contrast, ProDMP directly computes the closed-form solution of the second-order ODE as the position trajectory. This closed-form solution involves two coefficients, $c_1$ and $c_2$, corresponding to the complementary functions. From the fundamentals of solving linear ODEs, these coefficients can be uniquely determined given two initial conditions. In the Appendix A, Equation 19, we present the form of their solutions.

Regarding the basis functions, ProDMPs identify reusable terms, specifically the position and velocity basis functions, denoted by \Phi(t) and \dot{\Phi}(t), respectively. These are visualized in Fig. 10b in Appendix A. The mathematical representation is discussed in Equation 12 and Equation 14-16 of Appendix A.

More detailed description of the training/val/test split should be added. Specify how trajectories are divided between training, validation, and test sets. What are different between training and test? Clarify if test trajectories involve different objects, material properties, or initial conditions than training trajectories. In the limitation part, it is claimed ''We currently consider each trajectory as a task, and require initial states of this trajectory as a context set during inference.”

We thank the reviewer for the valuable feedback. We have updated Appendix D to provide more details about the training, validation, and test split. For the test splits, most tasks so far have involved in-distribution data, with variations in starting positions and collider trajectories, but with material properties either identical to or interpolated from those used in training.

To evaluate the model's generalization capabilities, we created a new data split in the Planar Bending task. In this split, the Young's Modulus values used for training ranged between 60 and 500, while the test set included values from [10, 30, 750, 1000], representing a clearly out-of-distribution scenario. Results from these experiments, included in the appendix, demonstrate that M3GN significantly outperforms both MGN and MGN (Oracle) in this setting, indicating strong generalization to out-of-distribution material properties.

We thank the reviewer for their thoughtful assessment and for recognizing the strengths of our work. The reviewer has raised important questions regarding the role of ProDMP in generating smooth trajectories and the contribution of the meta-learning framework to simulating new scenarios. Below, we provide detailed explanations to address these points.

Responses to Specific Concerns

Some methodology details are unclear, especially in the "Probabilistic Dynamic Movement Primitives" section and "Meta-Learning and Graph Network Simulators.”

We appreciate the reviewer's valuable feedback and have made several clarifications and improvements in response. We provide a detailed presentation of the ProDMP approach with its priors works in the appendix. Furthermore, we have reworked and improved the "Meta-Learning and Graph Network Simulators" section to enhance its clarity and ensure a more comprehensive presentation of our methodology.

How does ProDMP generate smooth trajectories based on the predefined conditions of the initial state? Please give detailed justification and explanation.

We thank the reviewer for the insightful question regarding how ProDMP generates smooth trajectories from predefined initial state conditions. Below, we provide a detailed explanation, the mathematical background can be found in Appendix A of the revised paper.

ProDMPs generate smooth trajectories by extending the concept of DMPs, which are designed to generate smooth motion trajectories for robots or other systems by defining a dynamical system that can be controlled using a set of parameters. The reason this system generates smooth trajectories is that the acceleration and velocity are continuously linked through the ODE. This mathematical formulation prevents sudden jumps in acceleration or velocity, which are what cause jerky, non-smooth motion.

To ensure that the trajectory starts from a specific initial state (i.e., a predefined position and velocity), ProDMPs use a technique to adjust the trajectory parameters such that the trajectory’s position and velocity match the given initial values. These adjustments are made through the introduction of special coefficients, which are computed based on the desired starting position and velocity. This guarantees that the trajectory begins smoothly at the specified initial state.

ProDMPs use a set of mathematical functions, known as basis functions, to define the shape of the trajectory. These functions are predefined and do not change during the learning process, allowing for computational efficiency. The trajectory is built by combining these basis functions with learned parameters that control the trajectory’s final shape. Because the basis functions are continuous and differentiable, they ensure that the trajectory evolves smoothly over time.

Could the author provide a detailed explanation of how a meta-learning problem can contribute to simulating new scenarios?

Meta-learning, often referred to as "learning to learn," is particularly effective for tasks where a model needs to generalize from one or more previous experiences to adapt to new, previously unseen situations. In the context of simulation, meta-learning enables the model to efficiently leverage past knowledge and quickly adapt to new simulation scenarios.

Meta-learning focuses on teaching models to generalize from prior tasks to new ones by learning shared representations or patterns across various tasks. For simulation, this means the model can learn general dynamics of deformable objects or interactions between objects and environments, rather than needing to learn from scratch for each specific scenario.

The model does not memorize specific tasks but learns a more abstract understanding of how to adapt its behavior to different environments, thus allowing it to simulate a wide range of new scenarios.

In traditional simulation methods, creating accurate models for each new scenario requires substantial computational resources and time. With meta-learning, once the model has learned how to adapt to new contexts, it can perform these adaptations in a more computationally efficient manner, even for complex simulations of deformable objects and their interactions. This could significantly reduce the time and resources needed for new simulation tasks, making it more scalable and efficient.

Concluding Remarks

We hope that the additional details and improvements make our contribution clearer and more accessible. We are grateful for the reviewer’s suggestions, which have helped us strengthen the manuscript, and look forward to any further feedback or questions that may arise.

Additionally, splitting single data points into multiple input-output sets seem to increase the effective amount of training data for M3GN, potentially creating an unfair comparison with MGN which use less training data.

We thank the reviewer for their comment. Both M3GN and MGN use the same underlying ground truth trajectories for training. While MGN processes each time step independently, M3GN operates on sequences of varying lengths, reflecting the differences in the training paradigms of the two methods. However, there is no inherent advantage or disadvantage to either approach in terms of the amount of training data used. Furthermore, to ensure a fair comparison, both methods are trained on the same hardware for the same amount of time.

9. The authors do not specify how material properties are incorporated. Also, it is unclear whether the test data involve material properties that are in-distribution or out-of-distribution relative to the training data. Providing this information is crucial for evaluating the model's generalization capabilities.

We appreciate the reviewer for pointing out this detail and are happy to provide the requested information. Material properties are explicitly incorporated only in the MGN (Oracle) baseline, as our approach assumes that material properties are unknown and must be inferred from the context. In the Oracle baseline, material properties are added as a global node feature.

For the test splits, all tasks so far involve mainly in-distribution data, with variations in starting positions and collider trajectories, but with material properties that are either identical to or interpolated from those used in training.

To evaluate the model's generalization capabilities, we created a new data split in the Planar Bending task and evaluated all methods on this. Here, the Young's Modulus values used for training ranged between 60 and 500, while the test set included Young's Modulus values from [10, 30, 750, 1000], representing a clearly out-of-distribution scenario. Results from these experiments, included in Figure 5, show that M3GN significantly outperforms both MGN and MGN (Oracle) in this setting, demonstrating strong generalization to out-of-distribution material properties.

We updated the paper to more clearly provide information about the datasets in Appendix D.

10. The authors mention that material node features are not added to M3GN. Given that these features enhance MGN's performance, it would be useful to understand the rationale for this exclusion and perform related ablation study.

While additional material information naturally improves prediction performance, it is not often available in realistic scenarios. As an example, fine-tuning a learned dynamics model from sensory data only provides geometry information and sensoric information, but not the internal task properties of the dynamics and the material. As such, M3GN is built around the idea of inferring a latent belief over this information, essentially learning to predict material information from a small context of observed behavior. We thus provided MGN (Material) as an “upper-bound” baseline with perfect knowledge about the material. We thank the reviewer for the valuable comment.

11. Although the authors mention other methods in related work besides MGN, these methods are not included in the baselines. Some of these methods have better accuracy and efficiency. Including these additional baselines would provide a clearer view of M3GN’s comparative performance.

We extend the related work section and included also more recent graph network simulators. To further strengthen the empirical evaluation, we compared our model against 2 more baselines. The first one is the requested History MGN[1] method, which includes velocities of previous steps as input to the GNN. The second method is the “Equivariant Graph Neural Operator”[4] baseline. We will update the paper with the results, however due to the short time of the rebuttal, this will take time until Wednesday.

12. Will the data used in this study be publicly available? Making the dataset accessible would facilitate further research and replication studies.

We will provide the full codebase and all used datasets upon acceptance of the paper.

We thank the reviewer for their valuable insights and detailed suggestions, especially in relation to textual clarifications and the inclusion of additional evaluations. We ran further trainings and evaluations regarding additional baselines and out-of-distribution testing. Additionally, we revised the paper to clarify the methodology and to provide additional visualizations to cover all requested issues. We provide a detailed response to the individual concerns raised by the reviewer.

Responses to Specific Concerns

1. The model's architecture is not clearly explained, and it is unclear why certain modules are necessary. For example, from the results, it seems that MGN, even without history information, can surpass M3GN in performance. This raises questions about the value of incorporating historical information in M3GN. Moreover, the experimental results do not clearly demonstrate the necessity or advantages of using a meta-learning scheme. A thorough analysis on how meta-learning benefits model performance would be valuable, including ablation studies comparing model performance with and without meta-learning.

We thank the reviewer for the feedback and provided additional explanations to the model’s architecture in the revised paper. The historical information lets our method build an accurate latent belief of the object dynamics for each meta-task, allowing for a more accurate simulation. This is shown in Figure 7, where various meta-learning methods outperform methods which do not incorporate the historical context. The ablation in Figure 7 also shows that the combination of meta-learning and parameterizing the trajectory with movement primitives is vital for exact simulations. We would like to highlight that the MGN baseline is only better on small context sizes for the Tissue Manipulation experiment. On all other experiments and context sizes, M3GN clearly outperforms MGN.

2. The authors claim that the baseline MGN does not incorporate historical information, which appears inaccurate. In certain datasets, MGN does include history. For a fair comparison, the MGN baseline should also be evaluated with historical data to assess its impact on performance.

We follow the original MGN paper [1] for our baseline implementation. The paper states, e.g., in the description of Figure 5 that “limiting history size increases accuracy by preventing overfitting”, hence the reason why we have not considered it. To obtain a complete view and to follow the request of the reviewer, we are running the experiments for the MGN with history information and expect the results on Wednesday. We will update the paper accordingly.

3. The results section only reports the average MSE across all time steps. It would be helpful to provide a comparison of MSE over the number of prediction steps, as this would give insight into the model's performance stability over time as claimed in the paper.

We thank the reviewer for the suggestion. To address this, we now include Figure 13 and Figure 14 in the Appendix, reporting the MSE over timesteps for all the tasks. These figures provide a detailed comparison of the models' rollout stability across all timesteps. As shown, autoregressive methods like MGN and MGN (Oracle) suffer from error accumulation, leading to higher MSE as the time progresses. In contrast, our M3GN demonstrates significantly better stability, with much lower MSE across timesteps due to the trajectory representation leveraged by the ProDMP method. This supports our claim regarding the improved stability of our approach.

4. Based on Figure 3, the proposed M3GN method does not appear to use ground truth collider information. If this is the case, does the collider state being predicted by the mode? How accurate is the collider state prediction, especially when history steps are limited? Additionally, including collider ground truth (as in MGN) is actually intuitive and makes sense, as the primary goal of developing a simulation model is to understand how a solid deforms under varying contact forces and obstacle displacements. Predicting these external forces may not be necessary for achieving this objective.

We thank the reviewer for their thoughtful question. The proposed M3GN method does not predict collider states. Instead, we use the collider trajectory during context timesteps and include only the last future collider position as a feature during prediction. This approach has proven effective for all tasks in our experiments, enabling accurate deformation predictions without unnecessary architectural complexity.

In preliminary experiments, we explored incorporating additional future collider positions, but found that it did not result in improved performance. However, we acknowledge that other tasks with different dynamics might require more extensive use of collider information. We addressed this in the paper, discussing how M3GN can adapt to incorporate additional collider states if needed.