SINGER: Stochastic Network Graph Evolving Operator for High Dimensional PDEs

We sincerely thank the reviewer for the insightful and encouraging feedback. We are pleased that the reviewer recognizes the novelty of our proposed SINGER framework for learning the evolution operator of high-dimensional partial differential equations (PDEs). Below, we provide detailed responses and clarifications to address the reviewer’s comments and questions.

W1: Baseline is not SOTA. It would be nice to see a hypernetwork of PINN or DeepONet.

A1. The problem of baselines can be discussed in three folds:

• SOTA Baselines

  1. NODE (Gaby & Ye, 2024, see reference [1]) is the SOTA for neural operators targeting high-dimensional PDEs, to the best of our knowledge. Currently, no studies have directly applied low-dimensional PDE neural operators (e.g., DeepONet or FNO) to high-dimensional problems. This is primarily due to their reliance on substantial reference data for training, with the required data volume increasing exponentially with spatial dimensions (the so-called curse of dimensionality). The training data is typically generated using numerical solvers, but high-dimensional PDEs lack accurate and efficient numerical solvers, making data generation challenging. Even if sufficient training data could be generated, the training time and memory requirements would be prohibitively large.
  2. PINN is not a valid baseline of neural operators. Well-recognized neural operator papers like DeepONet[2] and FNO[3] do not compare PINN as a baseline. The main reason might be that PINNs are not operators by definition. Since a PINN trained on one initial condition does not generalize to other conditions, i.e. it needs re-training for each new input. In addition, the PINN solving is much more time-consuming than a neural operator, making the comparison impractical and unfair. for example, the DeepBSDE solver [4] (a variant of PINN) solves a high-dimension PDE in 1 minute, while our trained SINGER solves it in about 0.1 seconds.

• Hyper-PINN-Nerual-Operator (PINO) To close the gap, in our paper, we designed a tailored Hyper-PINN-Nerual-Operator (PINO) as an additional baseline for the high-dimensional PDEs. Compared to standard neural operators, PINO introduces two major advancements. First, it replaces data loss with physics-informed loss, removing the dependence on training data. Second, it employs a hyper-network to generate parameters for a sub-network, which then produces a parameterized solution. This approach avoids directly outputting solutions at spatial sampling points, thus mitigating the exponential growth of network size with increasing spatial dimensions. These extensions make network training feasible.

• PINO Performance Although PINO is a feasible model, it falls short of SOTA theoretic support and empirical performance. From a model design perspective, PINO does not adhere to the semigroup assumption, whereas our proposed SINGER does (see also the direct evidence from Table 5). In terms of performance, experimental results on the Heat and HJB problems (Table 2) indicate that PINO consistently underperforms compared to NODE and our SINGER across all tested dimensions. In some cases, it even produces entirely incorrect results. For example, for the HJB problem with d=15 and d=20, the relative error of PINO approaches or exceeds 1, comparable to the performance of a dummy model producing zero outputs. These results suggest our adapted PINO still lags behind the available peer methods.

Q2: What are the benefits of stochasticity? Need ablation on the noise level.

A2. Moderate noise, compared to no noise, can enhance both stability and performance. Theoretical support for this claim is provided by Theorems 2–4, while experimental evidence is presented in Table 2 and Figure 4. Adding noise accelerates convergence and improves the final results, whereas the absence of noise often leads to training collapse.

However, the magnitude of noise does not improve performance indefinitely; excessive noise can be detrimental. In our paper, we fixed the noise level at 0.1. To further investigate, we conducted additional experiments testing the relative error of the heat equation under different noise levels (see Table A2 below). The results indicate that as the noise level increases from zero, the error initially decreases and then begins to rise. This behavior likely reflects a trade-off between the stability introduced by noise and the information loss it causes.

Table A2. SINGER Relative Error on Heat Equation under various noise levels.

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the novelty of our approach in solving high-dimensional time-dependent PDEs using a neural network model driven by stochastic differential equations and graph neural networks. Below, we carefully address the reviewer’s specific questions and concerns:

Q1. What is the PDE to be solved? Is there any restriction on the class of PDEs?

A1. We have updated the problem setting in Eq. 3 in the new draft. The class (e.g. elliptic, hyperbolic, and parabolic class) of PDEs is not restricted. Currently, we only consider time-dependent PDEs with smooth solutions. In the future, we will extend the algorithm to discontinuous solutions by applying weak form residual.

Q2. Clarify on the sub-network U and G(V,E). If it is a fully connected network (FCN), does it mean E is always 1 between any pair of vertices? How do you set it a priori? How critical is the performance under variations of sub-networks (what if you use the entire full network)?

A2. The questions of G and U can be answered in three parts:

• Construct G from U. The graph nodes V are parameters of U, and graph edges are connections of parameters. Two parameters are connected if they handle the input and output of the same intermediate variable (i.e. neuron). The parameters in the same layer is always not connected. For a toy example, consider a 1-width network $U(x)=c\sigma(ax + b) + d$, the resulting graph G is $a \to b \to c \to d$, which degenerates to a linked list. This example is a special case of FCN, and the resulting E is not always 1, i.e. G is not a full graph. In fact, one can observe that the G is always a bipartite graph.

• Configure U a priori. For Heat and HJB equations, we follow the network configuration in [1], which is a 2-layer FCN with boundary value function encoded in activation function (Eq. 14 and 15). For the other 6 PDEs, we use 2-layer FCN with tanh activation. The configuration of FCN is for simplicity.

• Sensitivity. To show the sensitivity of network configuration, we tested the Heat equation as shown in Table A2. One can observe that the -1 Layer (i.e. single layer FCN) is worse than the baseline, while +1 Layer and +10/-10 Width all outperform the baseline, indicating that our current network configuration can be further optimized. The variations above are all bipartite graphs. We also tested the full graph model you mentioned, but it only produces degenerated results, as shown in the last column in the table.

Table A2. Sub-network configure sensitivity

Q3. How to specify noise N? Is it based on the Lipschitz constant L of V?

A3. Setting $N > \sqrt{2L}$ as you mentioned is a theoretical method for setting noise level. However, in practice, it is challenging to estimate $L$ since it requires a high-dimensional search and changes during training. In our experiment, we set the noise level N=0.1 constant for simplicity. Below we provide a comparison of different settings of $N$. It is clear that the relative error first decreases and then increases as the noise level increases. This phenomenon might result from a trade-off of stability and information loss due to noise.

Table A3. SINGER Relative Error on Heat Equation under various noise levels.

Q4. Fairness of NODE and PINO baselins. How does SINGER compare to high-dim PDE solvers such as DeepRitz and PINN?

A4. The baselines can be discussed in two folds.

• PINNs. PINN, including its high-dim variants such as DeepRitz and DeepBSDE, is not a valid baseline of neural operators. Well-recognized neural operator papers like DeepONet[2] and FNO[3] do not compare PINN as a baseline. The main reason might be that PINNs are not operators by definition. Since a PINN trained on one initial condition does not generalize to other conditions, i.e. it needs re-training for each new input. In addition, the PINN solving is much more time-consuming than a neural operator, making the comparison impractical and unfair. for example, the DeepBSDE solver [4] solves a 20-dimensional PDE in about 1 minute, while our trained SINGER solves it in about 0.1 seconds.

• Neural Operators. NODE [1] is the SOTA for neural operators targeting high-dimensional PDEs, to the best of our knowledge. In addition, we designed a baseline of Hyper-PINN-Nerual-Operator (PINO), which can be regarded as a special kind of PINN. Although PINO is a feasible model, it falls short of SOTA theoretic support and empirical performance. From a model design perspective, PINO does not adhere to the semigroup assumption, whereas our proposed SINGER does. In terms of performance, experimental results on the Heat and HJB problems (Table 2) indicate that PINO consistently underperforms compared to NODE and our SINGER across all tested dimensions.

Q5. More theoretical work is needed for the semigroup property in a weaker sense (in distribution instead of almost surely). Does Table 5 suggest that the semigroup is weakened after training?

A5. Thank you for the valuable suggestion. In the updated draft, we have clarified the strong (pathwise or almost surely) and weak (identical distribution) stability in Definition 1, and prove that SINGER satisfies both strong and weak stability in Appendix B. The proof follows naturally from the properties of the Itô formula and Itô integral.

Besides, Table 5 seemingly suggests that the semigroup is weakened in the numerical sense after training. However, the relative error only increases to 1e-10 (1 out of 10 billion), which is usually regarded as numerically insignificant.

Q6. Minor: typo of $u$ and $U$.

A6. Thank you for pointing out that, we have fixed all typos about $u$ and $U$.

Reference:

[1] Gaby, Nathan, and Xiaojing Ye. "Approximation of Solution Operators for High-dimensional PDEs." arXiv preprint arXiv:2401.10385 (2024).

[2] Lu, Lu, et al. "Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators." Nature machine intelligence 3.3 (2021): 218-229.

[3] Li, Zongyi, et al. "Fourier neural operator for parametric partial differential equations." arXiv preprint arXiv:2010.08895 (2020).

[4] Han, Jiequn, Arnulf Jentzen, and Weinan E. "Solving high-dimensional partial differential equations using deep learning." Proceedings of the National Academy of Sciences 115.34 (2018): 8505-8510.

As we near the end of the discussion period, we would like to thank the reviewer again for the comments and provide a gentle reminder that we have posted a response to your comments. May we please check if our responses have addressed your concerns and improved your evaluation of our paper?

We sincerely thank the reviewer for the detailed and thoughtful feedback. We are pleased that the reviewer recognizes the importance of addressing the challenges in solving high-dimensional partial differential equations (PDEs) and appreciate our proposed SINGER framework for tackling these issues. Below, we provide detailed responses and clarifications to address the reviewer's comments.

Q1. The meaning and importance of 3 assumptions

A1. These assumptions stem from the fundamental properties of PDEs and neural networks. Existing PDE neural solvers do not inherently satisfy these properties (see Table 1). In contrast, we explicitly encode these assumptions into the network, allowing them to serve as prior knowledge to help the network learn more accurately.

• Graph Topological Property
  Neural network solvers for high-dimensional PDEs typically parameterize the target function $u$ using a neural network $u_\theta$. The internal computation of $u_\theta$ forms a graph where the parameters $\theta$ represent the nodes. For general network architectures, the corresponding graph is invariant to permutation (re-indexing of nodes). Additionally, random removal of a node, such as through dropout, does not drastically alter the network output. Networks that do not follow this assumption tend to be sensitive to input noise (see the comparative experiments in Table 5). However, when solving PDEs, we desire that the solution be robust to noise, which motivates encoding this property into the model.

• Semigroup Property
  This property arises from the physical interpretation of PDEs, where the time evolution of the solution is independent. Similar to the Markov property, the future state depends only on the current state, with no memory of the past. Encoding this prior to the network can prevent it from learning "non-physical" solutions.

• Stability
  Stability is another crucial property that originates from the well-defined nature of PDEs. In general, one typically considers solving well-posed PDEs, i.e., those for which a solution exists, is unique, and continuously depends on the initial and boundary conditions. The stability we define refers to the continuous dependence of the PDE solution on the initial condition. Stability is of paramount importance in both theoretical analysis and numerical solutions of PDEs. For example, in our experiments, models without stability would collapse during training (see Table 2).

Q2. Writing Modification

A2. We rewrite the contribution (section 1), the motivation (section 3), and the table row name (table 3), according to your suggestions. The new version of the draft is uploaded, and the modified parts are highlighted in blue.