Tensor methods to learn the Green's function to solve high-dimensional PDE

Thank you for the valuable comments.

• For the first experiment, we used same number of Monte-Carlo sample for all models, hence, Table 1 shows same sample size for DecGreenNet and DecGreenNet-TT. If you look at the Table 2 and Table 3, you can see that DecGreenNet-TT can learn with a large number of Monte-Carlo samples with higher accuracy and lower training times for higher-order PDE compared to DecGreenNet. This is possible since DecGreenNet-TT learns from partitions of dimensions with tensor-products via tensor-train decomposition instead of storing a large number of Monte-Carlo samples in as input for DecGreenNet.

• Yes, it is possible to approximate $\log(x_1 + x_2)$ and $\sqrt{x_1 + x_2}$ by a polynomial of $x_1$ and $x_2$. However, such expansions will require many neural networks to construct the tensor-train structured Green’s functions as in Theorem 1. Such approach can be computationally expensive and may lead to low accuracy. The directly separable functions as given in the paper can be easily integrated with the proposed model.

• We do not have a theoretical analysis for the tensor-structured low-rankness for Green’s function. Establishing tensor low-rankness is a future work and in this paper we have made an assumption that it exists. We point out that robust learning of PDEs in experiments justifies our claim that low-rankness exists for the Green’s function.

• MOD-net is computationally expensive to train for 2D PDE problems due to approximation of the Monte-Carlo integral approximation as demonstrated in [1]. The low-rank decomposition of the Green’s function has shown to improve the training times with DecGreenNet. In this paper we further improve on the DecGreenNet with tensor-structured low-rank decomposition and tensor-product grids (inspired by sparse grid and Smolyak grid [2])

• Training is stable for the problems we have used in the paper.

[1] Wimalawarne, Kishan, Taiji Suzuki, and Sophie Langer. "Learning Green's Function Efficiently Using Low-Rank Approximations." arXiv preprint arXiv:2308.00350 (2023)

[2] Hans-Joachim Bungartz and Michael Griebel. Sparse grids. Acta Numerica, 13:147–269, 2004.

Thank you for the valuable comments.

• We extended existing Poisson problems to higher-orders in our experiments since we could not find a suitable higher-order PDE problem in the existing machine learning literature. Does the reviewer has any suggestions?

• For N=4, we have used DecGreenNet-TT-C with combinations of 2 dimensions and from each combination we made 200 random partitions, hence, the total number of simulated samples are 200^2. All the other orders we used DecGreenNet-TT.

• In the Table 2, for N=2,3 we have used DecGreenNet-TT since we do not combine any dimensions. This can be identified by the number of $T_{*}$ networks. For N=4, we combined dimensions by pairs, so we only have 2 networks of $T_{1}$ and $T_{2}$, hence, we used the DecGreenNet-TT-C.

• We will include suggested references in the paper and update the relationships to the proposed method.

• We conducted further experiments with fixed equal spaced grid partitions for homogeneous Poisson problems with N=2 and N=3 using DecGreenNet-TT with similar networks structures used in Table 2. It shows that our proposed method performs well with different grid constructions.

N | Error | Time (sec)

2 | 0.006 | 51.97

3 | 0.205 | 146.21

Thank you for the valuable comments.

• Method such as H-matrix and butterfly decomposition are indeed used in solving PDE problems with Green’s function or Kernels [1]. In our understanding, for Green’s function formulation as ours can be formulated with Galerkin methods or collocation methods (Page 117 [1]) and solved as a linear system where matrix inverse approaches can be employed as suggested by the reviewer. If we parameterize the Green’s function with a neural network and apply Galerkin methods or collocation methods, then such model will not be equivalent to the PINNs framework. We are not certain, however, such models may be more computationally expensive to train neural networks using back-propagation due to recursive multiplications used in methods such as H-matrix. We agree that our proposed methods of tensor-structured decomposition of the Green’s function does not stem from existing theory. Our objective in this paper is to propose a model using tensor-train based decomposition with combination of sparse grid and Smolyak grid [3] to parameterize the Green’s function and support it with experiments. We think that analytical methods in [2] can be extended to tensor structured decomposition of the Green’s function, which we propose as a future research.

• Regarding notations and writing, we will update the writing in the revised version.

• Our referenced PDE models [4] and [5] have not considered the noise in PDE experiments. Hence, we also did not consider noise based PDE problems.

• In this paper, we wanted to propose the tensor-train model to parameterize the Green’s function and demonstrate its scalability with to higher-order PDEs. Hence, we used simple PDE problems to make our case.

[1] Mario Bebendorf, Hierarchical Matrices, 2008

[2] Bebendorf, M., Hackbusch, W. Existence of ℋ-matrix approximants to the inverse FE-matrix of elliptic operators with L ∞-coefficients. Numer. Math. 95, 1–28 (2003

[3] Hans-Joachim Bungartz and Michael Griebel. Sparse grids. Acta Numerica, 13:147–269, 2004.

[4] Zhang Lulu Luo, Tao Zhang Yaoyu, E Weinan, John Xu, Zhi-Qin, and Ma Zheng. Mod-net: A machine learning approach via model-operator-data network for solving pdes. CiCP, 2022

[5] Yuankai Teng, Xiaoping Zhang, Zhu Wang, and Lili Ju. Learning green’s functions of linear reaction-diffusion equations with application to fast numerical solver. In MSML, 2022

Thank you for the valuable comments.

• One of the baseline methods we considered was MOD-Net, however, it requires a considerable computation time for higher-order PDEs. Hence, we did not consider MOD-Net as a baseline methods. For reference for the 2D Poisson problem we refer to the [1]. We also considered PINNs as a baseline methods for homogeneous Poisson problem, which we list below.

N | Network structure | $\lambda_b$ | Error | Time (sec)

2 | [2, 512, 512, 512, 1] | 10 | 0.3428 | 13.18
3 | [3, 512, 512, 512, 1] | 1 | 3.6025 | 9.60
4 | [4, 512, 512, 512, 512, 1] | 1 | 36.71 | 13.59

the above results show that PINNs perform weaker compared to the proposed methods.

• There is not specific way that we can use for combining dimensions to find N'. We believe that we have select the best combination of dimensions by experiments. Hence, the the combinations of the dimensions and N' can be considered as hyperparameters.

[1] Wimalawarne, Kishan, Taiji Suzuki, and Sophie Langer. "Learning Green's Function Efficiently Using Low-Rank Approximations." arXiv preprint arXiv:2308.00350 (2023)