Neural Symbolic Regression of Complex Network Dynamics

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: limited innovation because of similarity to AI-Feynman and SymDL

R1: Our paper introduces a novel coordinated genetic search algorithm that leverages the strengths of both neural network interpolation and surrogate model referencing to uncover symbolic expressions representing complex network dynamics. While neural network interpolation and surrogate model referencing have been utilized separately in methods such as AI-Feynman and SymDL, our approach uniquely integrates these components in a synergistic manner. This integration, achieved through our coordinated genetic search algorithm, is the primary innovation of our work.

W2: more complex scenarios such as neuronal, genetic, social, and coupled oscillator dynamics & multi-dimensional dynamics

more complex scenarios: The dynamics in our current experiments already encompass the scientific domains you mentioned. For example, the Wilson-Cowan Neural Firing Dynamics and the FitzHugh-Nagumo Dynamics (newly added in Appendix B.6 of the revision) represent complex dynamics in neuroscience. The Lotka-Volterra Population Dynamics are applied in modeling genetic evolution, while the Susceptible-Infected-Susceptible Epidemics Dynamics are used by social scientists to model the spread of rumors. Kuramoto Oscillators are a type of coupled oscillator dynamics.

Multi-dimensional network dynamics: We add a two-dimensional network dynamics experiment of the FHN model in Appendix B.6 of the revision. The proposed method can handle the multi-dimensional network dynamics after extending the existing genetic search library, gplearn, to support vector-valued functions.

Questions

Q1: computational efficiency of GA compared to TP-SINDy in the symbolic regression component

A1: We have reported the running time of our method in Appendix B.2. While TP-SINDy completes in approximately 4 minutes, PI-NDSR requires 50 to 60 minutes. Although running time is not the primary strength of our method, this should not be considered a drawback. The main objective of symbolic regression is to uncover and better understand the underlying dynamics of a system, where accuracy is of utmost importance. Sacrificing some computational efficiency for more accurate symbolic expressions is a worthwhile trade-off.

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: rationality of the method design & strong assumption of decoder

R1:

Rationality of the method design: The design of the encoder and decoder is motivated by neural algorithmic reasoning [1]. In this paper, we regard the dynamics of the network as an algorithmic reasoning task, i.e., an algorithm with an ODE. So we incorporate various techniques, i.e., algorithmic alignment [1] and encoder-decoder paradigm, into the neural network design.

Strong assumption of decoder:

• MLP is expressive enough to satisfy the assumption: The decoder's additivity assumption can be met by a general MLP since the input and output of the decoder are confined to some specific subspace. Within this subspace, the additivity assumption is relatively weak, allowing the decoder to be trained accordingly.
• MLP satisfies the assumption empirically: We conduct experiments to show that the decoder in our model can satisfy the assumption empirically with SIS dynamics. Since the states of SIS dynamics are bounded in range $[0,1]$, we randomly sample two values $x_1$ and $x_2$ from $[0,1]$ and map them to the hidden representation $\mathbf{h}_1$ and $\mathbf{h}_2$ with encoder. Then we calculate the MSE between $\mathsf{Dec}(\mathbf{h}_1 + \mathbf{h}_2)$ and $\mathsf{Dec}(\mathbf{h}_1) + \mathsf{Dec}(\mathbf{h}_2)$. The result is $0.0021$, which is very small and means that the decoder can satisfy the assumption.

[1] https://thegradient.pub/neural-algorithmic-reasoning

[2] What Can Neural Networks Reason About?

W2: The contribution lacks clear clarification.

R2: Our contributions are summarized in the following two points:

• Using both the interpolated denoised trajectories and references of node/edge dynamics from physically inspired neural dynamics for supervision, PI-NDSR proposes a coordinated genetic search algorithm to avoid overfitted expressions and improve the quality of symbolic expressions.
• Compared to previous methods, PI-NDSR has better recovery probability and smaller errors in the recovered symbolic dynamics.

To clarify the contribution, we conclude the contribution of our work at the end of the introduction of the revision.

W3: time cost analysis & one-dimensional dynamics.

R3: Time cost: We have reported the running time of our method in Appendix B.2. While TP-SINDy completes in approximately 4 minutes, PI-NDSR requires 50 to 60 minutes. Although running time is not the primary strength of our method, this should not be considered a drawback. The main objective of symbolic regression is to uncover and better understand the underlying dynamics of a system, where accuracy is of utmost importance. Sacrificing some computational efficiency for more accurate symbolic expressions is a worthwhile trade-off.

Multi-dimensional network dynamics: We add a two-dimensional network dynamics experiment of the FHN model in Appendix B.6 of the revision. The proposed method can handle the multi-dimensional network dynamics after extending the existing genetic search library, gplearn, to support vector-valued functions.

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: lacking evaluation: filtered evaluation metric

R1: We choose MSE restricted to correctly recovered skeletons because the baseline methods often exhibit large MSE when recovering incorrect skeletons. Filtering out these formulas allows the baseline to achieve comparable performance and also reduces the variance of the results.

W2: Brief conclusion, limitation, and future work

R2: We have emphasized the key findings and improvements over existing methods in the conclusion of the revision. We also added a discussion of the impact statement and detailed future work in Appendix C of the revision.

Questions

Q1: What could be the potential issues when using the model for highly complex formulations?

A1: The highly complex formulations indicate a large search space for the genetic algorithm. Therefore, there should be a large population size and a large number of generations for the genetic algorithm to find the symbolic expressions. Since the fitness of our method is calculated based on the pairwise combination of node and edge dynamics, the fitness evaluation is computationally expensive and memory-consuming.

Q2: Report both filtered and unfiltered MSE.

A2: See R1.

Q3: Expand the discussion of the limitations.

A3: We have added a detailed discussion of the limitations in Appendix C of the revision.

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: novelty of interpolation method

R1: The novelty of our work lies in integrating a neural network-based interpolation method with a new genetic algorithm-based symbolic regression approach to recover symbolic expressions of complex network dynamics. Our method, PI-NDSR, introduces a novel genetic search strategy that leverages both the interpolation model and references of node/edge dynamics from physically inspired neural dynamics for supervision, effectively mitigating the risk of overfitting in the symbolic space.

W2: The sparsity issue remains partially unresolved, as the interpolation might overlook important variations in sparse data.

R2: Your concern is valid. Whether interpolation can truly address data sparsity depends on the specific data patterns. If the data has a lot of sudden changes, interpolation may not work well. In the complex network dynamics we studied, the data does not have many sudden changes, and interpolation can work well.

W3: The rationale for decoupling node and edge dynamics

R3: The rationale for decoupling node and edge dynamics is motivated by a theoretical paper [1] published in 2020 which studies the reasoning ability of DNN. This paper provides a mathematical framework explaining why certain architectures may be more suitable for particular reasoning tasks. The primary theorem (Theorem 3.6 of [1]) asserts that improved algorithmic alignment (Definition 3.3 of [1], structural correspondence between functions of a reasoning task and different modules in DNN) leads to enhanced generalization.

Our paper wants to learn neural dynamics with strong generalization ability. Based on algorithmic alignment, Eq.(1) provides a strong inductive bias for the neural dynamics design. The designed model, Eq. (2), is algorithmically aligned with Eq. (1), allowing our model to demonstrate strong generalization. With strong generalization, the decoupling of node and edge dynamics can provide better interpolation trajectories and references for the coordinated genetic search algorithm to recover symbolic expressions of complex network dynamics. For example, the node dynamics can be seen as the dynamics on a complex network with a single node. We added this explanation to Appendix C of the revision.

If you are interested, you can check out the blog [2] for model designs (e.g., the encoder-decoder design in our paper) and applications of algorithmic alignment.

[1] What Can Neural Networks Reason About?

[2] https://thegradient.pub/neural-algorithmic-reasoning

Questions

Q1: Can interpolation truly address data sparsity?

A1: See R2.

Q2.1: Reasons and impacts of the challenges in symbolic regression for complex network dynamics, e.g., noise, data sparsity

A2.1: Existing methods use a five-point approximation to numerically estimate the derivative $\dot{\mathbf{X}}(t)$, which can lead to large estimation errors in the presence of noise or when the sampling is sparse. Our method, on the other hand, uses a neural network to interpolate and denoise $\mathbf{X}(t)$ instead of $\dot{\mathbf{X}}(t)$, resulting in smaller errors. An example is provided in Figure 5 of Appendix B.5.

Q2.2: The motivation for decoupling node and edge dynamics lacks sufficient detail

A2.2: See R3.

Q2.2: The example of overfitting in symbolic space

A2.2: We provide an example of overfitting in Appendix B.4.

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: It is not clear if this method could scale to real-world datasets that have more complex dynamics, since there is only a single real-world dataset.

R2: Thank you for your suggestion. Our approach can indeed be applied to more real-world datasets, aiding scientists in understanding complex real-world systems. However, this often requires extensive data collection efforts. We are collaborating with experts in related fields to explore the application of our method to brain networks, transportation networks, and protein network data. Due to the inherent complexity of these tasks, we consider them part of our future work.

Questions

Q1: details about about training 4 networks

A1: We combine the four networks into a single network, which is trained end-to-end with $\min_{\theta}\sum_{v\in V, t\in \mathcal{T}}\|f_\theta (\mathsf{G}, \mathbf X(t_0), t)_v - x_v(t)\|_1$. We have added details to how the neural dynamics are trained in Section 4.1 of the revision.

Q2: typos in the paper

A2: Thank you for pointing that out. We have corrected this in the revised version.

We thank the reviewer for valuable comments and suggestions. We respond to the concerns raised by the reviewer as follows:

Weaknesses

W1: novelty of interpolation model

R1: The novelty of our work lies in integrating a neural network-based interpolation method with a new genetic algorithm-based symbolic regression approach to recover symbolic expressions of complex network dynamics. Our method, PI-NDSR, introduces a novel genetic search strategy that leverages both the interpolation model and references of node/edge dynamics from physically inspired neural dynamics for supervision, effectively mitigating the risk of overfitting in the symbolic space.

Questions

Q1: Have the authors attempted to explore solutions for the missing data problem?

A1: Current methods are unable to handle the missing data problem. In future work, we will explore potential solutions such as introducing latent variables into the network dynamics.

Q2: Suggestions for reducing this computational cost

A2: The computational cost of PI-NDSR is mainly from two parts, i.e., neural dynamic training and symbolic regression with genetic search. In the future, we will explore improving the computational efficiency of genetic search by incorporating more advanced symbolic regression methods, e.g., transformer-based and LLM-based symbolic regression methods.