Graph ODE with Factorized Prototypes for Modeling Complicated Interacting Dynamics

Q3. Missing related work: https://openreview.net/forum?id=B1lGU64tDr, where GNNs are also used to model the dynamics of interacting objects and similar auxiliary losses are designed for representation learning.

A3. Thanks for your comment. We have added the discussion with [4] as follows: "R-SSM [4] models the dynamics of interacting objects using GNNs and includes auxiliary contrastive prediction tasks to enhance discriminative learning while our study focuses on dynamical systems under potential parameter shift and develop a graph ODE framework with high expressivity and generalization capacity."

Q4. The authors claim that the mutual information estimator is trained adversarially, but I don't see any adversarial training in Appendix D. The authors should clarify this point.

A4. Thanks for your comment. Sorry for the typo. We have revised Appendix D by including: Step 11: Update $\tau'$ in our PGODE using gradient ascent; Step 12: Update other parameters in our PGODE using gradient descent.

Reference

[1] Yuan et al., AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting, ICCV 2021.

[2] Kipf et al., Neural Relational Inference for Interacting Systems, ICML 2018.

[3] Yildiz et al., Learning interacting dynamical systems with latent Gaussian process ODEs, NeurIPS 2022.

[4] Yang et al., Relational State-Space Model for Stochastic Multi-Object Systems, ICLR 2020.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Q2. GNN-based discrete-time models are not compared in the experiments. It would be interesting to see how the proposed model compares to these models. For example, the NRI model (Kipf et al., 2018) (where the Springs and Charged datasets are from) could be a good baseline.

A2. We have included the baseline AgentFormer [1], NRI [2] and I-GPODE [3] in our performance comparison. The results of these comparisons are presented below and our method outperforms the compared methods, which verifies the superiority of our method. We have updated the manuscript accordingly.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. The mixture-of-experts architecture is not well-motivated. While the authors introduce the mixture as a way to enhance the expressivity of the model, it is unclear to me what is the advantage over a single message-passing function that also takes the node representation and the global representation as input.

A1. Thanks for your comment. We have included a model variant PGODE w. Single, which takes the node representation and the global representation as input with a single message-passing function. The compared performance on the two datasets is recorded as below. From the results, we can observe that the full model performs better than PGODE w. Single, which shows that directly combining representation and the global representation cannot fully capture the complicated dynamics of every node. In contrast, our prototype decomposition can involve different GNN bases, which model diverse evolving patterns to jointly determine the individualized dynamics. This strategy can enhance the model expressivity, allowing for more accurate representation learning of hierarchical structures from a mixture-of-experts perspective. We have updated the manuscript accordingly.

Reference

[1] Ricky T. Q. Chen, Neural Ordinary Differential Equations, NeurIPS 2018.

[2] Kong et al., Sde-net: Equipping deep neural networks with uncertainty estimates, ICML 2020.

[3] Luo et al., Hope: High-order graph ode for modeling interacting dynamics. ICML 2023.

[4] Yuan et al., AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting, ICCV 2021.

[5] Kipf et al., Neural Relational Inference for Interacting Systems, ICML 2018.

[6] Yildiz et al., Learning interacting dynamical systems with latent Gaussian process ODEs, NeurIPS 2022.

[7] Yang et al., Relational State-Space Model for Stochastic Multi-Object Systems, ICLR 2020.

[8] Rhodes et al., Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization, NeurIPS 2021.

[9] Xu et al., Multi-objects Generation with Amortized Structural Regularization, NeurIPS 2019.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Lastly, we have included more recent literature in our related works as: "AgentFormer [4] jointly models both time and social dimensions with semantic information preserved. NRI [5] models interactions along with node states from observations using GNNs. I-GPODE [6] estimates the uncertainty of trajectory predictions using the Gaussian process, which facilitates effective long-term predictions. R-SSM [7] models the dynamics of interacting objects using GNNs and includes auxiliary contrastive prediction tasks to enhance discriminative learning."

Q3. What is the connection between maximum likelihood estimation and representing a distribution by its mean mentioned in Section 3.3 by sentence: Following the maximum likelihood estimation, here we solely produce …”

A3. Thanks for your comment. Here, we assume that the distribution of $\hat{x}^t_i$ follows an Gaussian distribution with variance $\sigma^2$ and maximizing the likelihood estimation can get $\min$ $\sum_r$ $\frac{ ||x^t_i-\hat{x}^t_i (r) ||^{2}}{2 \sigma^{2}}$, where $\hat{x}^t_i (r)$ denotes the r-th sample from the distribution. Minimizing this is equivalent to $\min \frac{||x^{t}_i-\mu^{t}_i||^{2}}{2 \sigma^{2}}$, where $\mu^{t}_i$ denotes the mean of $\hat{x}^t_i (r)$.

Q4. The ELBO part of the loss makes perfect sense. The way the other two terms are used in Equation 18 is also derived following well-justified desiderata. However, using the three terms together as part of a Bayesian inference objective is another game. What is the forward model implied by the eventual loss in Equation 18?

A4. Thanks for your comment. In Eqn. 18, the other two loss terms serve as a regularization mechanism using mutual information to constrain the model parameters. This strategy is popular in recent Bayesian inference models [8,9]. Therefore, our integration improves the predictive performance while maintaining a robust Bayesian inference framework.

Q5. How does the proposed method compare numerically against I-GPODE of (Yildiz et al., 2022)?

A5. Thanks for your comment. We have included the baseline I-GPODE in A1.

Q2. It is very surprising that despite the extreme similarity of the paper structure, even Figure 1 and Lemma 3.1, to the material reported in (Luo et al., 2023), the paper does not clarify the difference between the proposed method to this recent work. Likewise, the paper aims to solve the exact same problem as Yildiz et al., 2022’s I-GPODE, which even has a subsection about the disentanglement properties of their Graph ODE based interaction model. Yet the proposed method is not differentiated from it. To me even a numerical comparison to such recent work developed for the same exact purpose (interaction modeling) based on the same conjecture (disentanglement) is essential. Overall, I think the literature positioning aspect of the paper is extremely premature.

A2. Thanks for your comment. Firstly, HOPE [3] has been included in the performance comparison. Now we have included a detailed comparison as follows.

• Different Objectives: Our method primarily hones in on modeling dynamical systems in the presence of potential parameter shift. In contrast, HOPE does not encompass this specific challenge.
• Different Methodology: Our method introduces hierarchical context discovery with disentanglement to guide the prototype decomposition of individual nodes in modeling interacting dynamics while HOPE focuses on incorporating second-order graph ODE in evolution modeling.
• Different Performance: From the performance comparison, our proposed method performs much better than HOPE by over 47%.

Secondly, we have included the baselines AgentFormer [4], NRI [5] and I-GPODE [6] in our performance comparison. The results of these comparisons are presented below and our method outperforms the compared methods, which verifies the superiority of our method. We have updated the manuscript accordingly.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. While I appreciate the effort made in the paper to give theoretical guarantees, I have reservations about its correspondence to the main goal. Lemma 3.1. guarantees only that the ODE has a unique solution, which covers only the computational aspect. This is not a surprising, nor a crucial result. In machine learning approaches to time series modeling, the main challenge usually is not to design ODEs with well-defined solutions. It comes straightforwardly after the Lipschitz continuity assumption that is easy to satisfy when the function approximators are neural nets. The main challenge is rather the predictive power and generalization capacity of the developed models.

A1. Thanks for your comment. We agree that the predictive power and generalization capacity are more important, which have been empirically validated by our extensive experiments compared with state-of-the-art approaches. Our theoretical guarantee of well-defined solutions still has merit, which is put forward by the first NODE paper [1] and emphasized by the following works [2]. This theoretical guarantee of the existence and uniqueness would provide insight in designing stable network architecture [2]. However, this is only a supplementary to our core contribution, which is the design and application of powerful graph ODEs for modeling dynamics systems, as outlined in Sec. 1 of our paper.

Dear Reviewer,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the literature positioning, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.

Q1. On the flip side, the complexity of the proposed method is rather high. The integration of a mixture of experts not only elevates the intricacy but also augments the number of parameters, consequently escalating the training cost when juxtaposed with the baseline methods. A comparative analysis with an ensemble of baseline methods might present a fairer landscape for evaluation, given the escalated complexity and training cost.

A1. Thanks for your comment. We have included the comparison of computation cost as follows and we can observe that our method has a competitive computation cost. In particular, the performance of HOPE is much worse than ours (the performance increasement of ours is over 47% compared with HOPE), while our computational burden only increases a little. Moreover, both the performance and efficiency of I-GPODE are worse than ours.

Q2. The training cost is a pivotal factor for practical implementation. How does the training cost of the proposed method compare with that of the baseline methods? This comparison would provide a clearer understanding of the practical implications entailed by the proposed method.

A2. Thanks for your question. We have included the comparison in A1.

Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.

Dear Reviewer,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the complexity, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

Q3. The problem setup seems to sweep the most important thing under the rug. They talk about the problem of forecasting the dynamics of predicting the features, but what about continuously predicting the changes in the graph? Or are these all given? Aren't their observations discrete? Don't the graphs have to be updated continuously also, and how exactly is the graph changed at each iteration of the graph ODE? These and many questions are unanswered.

A3. Thanks for your comment. Here, although the observations are discrete, the underlying dynamics should be still continuous. Modeling continuous trajectories enables us to extrapolate and predict outputs at any given point in time, effectively bridging the gap between discrete observations and continuous underlying dynamics. As for graphs in physical simulations, the graph structures remain fixed, which are from real springs or electric charge effects following previous works [3,7]. In molecular simulations, the graph structures remain stable for a small period in reality, and therefore, we update the graph structure based on node position every $T$ timestamps. Modeling interacting dynamics with fixed or robust graphs is an important task, which has received extensive attention as in [3,4,5,6,7]. We have updated the manuscript accordingly.

Q4. I would recommend for the authors to rewrite the paper from scratch, not assuming anyone knows anything about agent dynamics or object contexts. Define every technical term, and fully specify the problem. Without a thorough rewrite it is very unclear what is going on.

A4. Thanks for your comment. We have rewritten the abstract, introduction and problem definition to make sure everything is clear. In particular, we have mainly rewritten the following parts.

• Abstract. This paper studies the problem of modeling multi-agent dynamical systems, where agents could interact mutually to influence their behaviors. Recent research predominantly uses geometric graphs to depict these mutual interactions, which are then captured by powerful graph neural networks (GNNs). ... The core of PGODE is to incorporate prototype decomposition into a continuous graph ODE framework, which is guided by contextual knowledge from historical trajectories.
• Introduction (First paragraph) Multi-agent dynamical systems are ubiquitous in the real world where agents can be vehicles and microcosmic particles. These agents could have complicated interactions resulting from behavioral or mechanical influences, which result in complicated future trajectories of the whole system. Modeling the interacting dynamics is a crucial challenge in machine learning with broad applications in fluid mechanics, autonomous driving and human-robot interactions.
• Introduction (Second paragraph) The core of PGODE lies in exploring disentangled contexts, i.e., object states and system states, inferred from historical trajectories for graph ODE with high expressivity and generalization.
• Problem Definitation. Given a multi-agent dynamical system, we characterize the agent states and interaction at the $t$-th timestamp as a graph $G^t = (\mathcal{V}, \mathcal{E}^t, **X**^t)$, where each node in $\mathcal{V}$ is an object, $\mathcal{E}^t$ comprises all the edges and $**X**^t$ is the object attribute matrix. $N$ represents the number of objects. Given the observations $G^{1:T_{obs}} = \{G^1, \cdots, G^{T_{obs}} \}$, our goal is to learn a model capable of predicting the future trajectories $**X**^{T_{obs}+1: T}$. Our interacting dynamics system is governed by a set of equations with system parameters denoted as $\xi$. Different values of $\xi$ could influence underlying dynamical principles, leading to potential shift in trajectory distributions. Therefore, it is essential to extract contextual information related to both system parameters and node states from historical observations for faithful trajectory predictions.

Q6. Exactly what sort of dynamics are you thinking of modeling and how does the graph changing over time get modelled in your system?

A6. Thanks for your question. Modeling interacting dynamics with fixed or robust graphs is an important task, which has received extensive attention as in [3,4,5,6,7]. Here, in physical datasets, we model the dynamics of objects interacted by connected springs or electric charge effects for trajectory predictions. Therefore, the graph is given and fixed. In molecular datasets, we model the dynamics of atoms and their interactions are related to the pairwise distance. In reality, the graph structures remain stable for a small period and we update the graph structure based on node position every $T$ timestamps. We have updated the manuscript accordingly.

Reference

[1] Das S, et al., Interpreting deep neural networks through prototype factorization, ICDM 2020.

[2] Zhao J, et al. ProtoViewer: Visual interpretation and diagnostics of deep neural networks with factorized prototypes, IEEE Visualization Conference 2020.

[3] Huang et al, Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations, NeurIPS 2020.

[4] Fang et al., Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting, KDD 2021.

[5] Cao et al., Efficient Learning of Mesh-Based Physical Simulation with Bi-Stride Multi-Scale Graph Neural Network, ICML 2023

[6] Han et al., Predicting Physics in Mesh-reduced Space with Temporal Attention, ICLR 2022

[7] Kipf et al., Neural Relational Inference for Interacting Systems, ICML 2018.

[8] Yuan et al., AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting, ICCV 2021.

[9] Yildiz et al., Learning interacting dynamical systems with latent Gaussian process ODEs, NeurIPS 2022.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Q5. Moreover, there are other agent based models like the agent former that the authors have not compared their method with.

A5. Thanks for your comment. We have included the baseline NRI [7], AgentFormer [8], and I-GPODE [9] in our performance comparison. The results of these comparisons are presented below and our method outperforms the compared methods, which verifies the superiority of our method. We have updated the manuscript accordingly.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. This paper is very hard to read and uses jargon that most ML people may not be familiar with. I believe this paper is trying to do not just interacting dynamics but actually agent-based modeling. Given this context, I am completely unclear as to what a "factorized prototype" is, the authors just use this term without definition and repeatedly explain the method this way. The other key feature of an agent based system that's not just in an interactive dynamic system (like a gene regulatory network) is that the graph consisting of agents changes with time based on the movements of the agents. This would be the key challenge to using a GNN/Graph ODE to model agent dynamics and this is not addressed here clearly despite defining the data as consisting of different graphs at different timepoints.

A1. Thanks for your comment. Firstly, let me clarify "factorized prototype", which refers to the decomposition of an agent's behavior within a system into several fundamental elements. In our context, each GNN within the graph ODE framework acts as a base model (a.k.a., prototype). The interaction dynamics of each agent are then represented as a weighted combination of these prototypes, which is formulated as our ODE function (Eqn. 10). This concept of "factorized prototype" has been utilized in several previous works [1,2]. However, to improve clarity, we have revised this terminology to "prototype decomposition," a more widely used term in the field of machine learning. We also rename our method as Prototypical Graph ODE (PGODE). Secondly, we agree that the evolution of graphs over time is a significant challenge in dynamical system modeling. However, our current research centers on predicting the trajectories from graph-structured data rather than inferring dynamical graphs, which is aligned with [3,4,5,6,7]. In particular, in Springs and Charged, the fixed graphs are given based on real springs or electric charge effects rather than node positions. We acknowledge the importance of graph inference in more complicated scenarios and plan to explore it in future work, which has been stated in the revised version of our paper.

Q2. Additionally the authors refer to some type of hierarchical representation which is again completely unclear. My best guess is that each object somehow evolves based on a combination of those GNN "factored prototypes" but that seems somewhat strange. Couldn't we just learn an individualized evolution based on a GNN with an MLP aggregation layer? What is the need for "factored" prototypes, I can at least see an argument for selecting a prototype (using a softmax or something of this sort).

A2. Thanks for your comment. To address your concern, we have included a model variant PGODE w. MLP, which combines a GNN with an MLP to learn the individualized dynamics. The compared performance on the two datasets is recorded as below. From the results, we can observe that the full model performs better than PGODE w. MLP, which shows that a straightforward MLP cannot fully capture the complicated dynamics of every node. In contrast, our prototype decomposition can involve different GNN bases, which model diverse evolving patterns to jointly determine the individualized dynamics. This strategy can enhance the model expressivity, allowing for more accurate representation learning of hierarchical structures from a mixture-of-experts perspective. We have updated the manuscript accordingly.

Dear Reviewer,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the presentation, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

Dear reviewers,

We sincerely appreciate your valuable feedback.

As the deadline for the author-reviewer discussion phase is approaching, we would like to check if you have any other remaining concerns about our paper.

We sincerely thank you for your dedication and effort in evaluating our submission. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors.