Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations

We appreciate the detailed suggestions, criticisms and endorsement of the reviewer. We address all of these below:

My primary concern is with the ReLU network approximation ... to ensure consistency between these sections.

Additionally, I noticed that the notation represents one step ... appropriate reference for the approximation result, specifically one that addresses shallow networks.

We appreciate your point. Indeed, the current version did not provide sufficient explanation of how the neural operator defined in Definition 1 represents Eq. (24), despite its importance.
To clarify, one step $\Phi_{N, \text{net}}$ in Eq. (24) corresponds not to a single-layer of the neural operators, but to a multi-layer. Specifically, when the neural network $F_{\text{net}} : \mathbb{R} \to \mathbb{R}$ has $L$ layers, $\Phi_{N, \text{net}}$ can represent a neural operator with $L+1$ layers. In this construction, the local operators $W^{(\ell)}$ and biases $b^{(\ell)}$ for $\ell = 1, \dots, L$ correspond to those in the neural network $F_{\text{net}}$, while the non-local operators $K^{(\ell)} \equiv 0$ for $\ell = 1, \dots, L$. At the $L+1$-th layer, $W^{(L+1)} \equiv 0$ and $b^{(L+1)} \equiv 0$, and the non-local operator $K^{(L+1)}$ corresponds to the integral operator used in Eq. (24).
We have added these detailed explanations following Eq. (9) in the revised version.

Another minor question is that, I think your remark ... the Green’s function of the linear equation.

We agree this sentence is not important. We removed it in the revised version.

We appreciate the detailed suggestions, criticisms and endorsement of the reviewer. We address all of these below:

Problem in the proof of Lemma 5. The main issue I currently ... is required at each layer; for each function evaluation.

We first empathize that our analysis does not have errors in the proof. Since our explanation was insufficient, we answer your question as follows:

Suppose that we have the integral

By considering the extension $\tilde{G} : [0,T] \times [0,T] \times D \times D \to \mathbb{R}$ of $G$ by

and expanding $\tilde{G}$ (regarding a function on $[0,T] \times [0,T] \times D \times D$) using basis function $\varphi: [0,T] \times D \to \mathbb{R}$ and $\psi:[0,T] \times D \to \mathbb{R}$, we can see that

where coefficients $c_{n,m}$ are given by

Note that coefficients $c_{n,m}$ should be independent of input functions, especially time $t$, because coefficients $c_{n,m}$ correspond to learnable parameters in non-local operators. This is why we adapted the integral from $0$ to $T$, instead of the integral from $0$ to $t$, in the formulation of our neural operators. We added some explanation in the beginning of Section 3 in the revised version.

Note that, due to the form of integral equation (P'), our neural operators need to take the integral with respect to time, which may be computationally expensive as your point. However, the previous work of Section 7.3 of [Kovachki et al, 2023] has employed the Fourier transforms with respect to time for computing FNOs, whose techniques might be useful for computing our neural operators. We added these discussions in Section 5 of the revised version.

1. The paper mentions that Theorem 1 avoids ... may still grow exponentially in $d$ ?

Our primary focus is not on avoiding the curse of dimensionality with respect to the spatial dimension $d$. Instead, we aim to control model complexity by providing effective bounds for the depth $L$, the number $H$ of neurons, and the rank $N$ when precision $\epsilon$ goes to zero. We have achieved this goal, ensuring that our rates do not grow exponentially. However, addressing the curse of dimensionality with respect to $d$ would also be an interesting direction for future works. In fact, the constant $C$ in Theorem 1 depends on $|D|$ (see Eq. (23)). For example, if $D$ is a hypercube with side length $R>1$, then then $|D| \approx R^d$, which grows exponentially as $d \to \infty$. We have noted this observation as a limitation of our work below Theorem 1 in the revised version.

2. Theorem 1 requires ... e.g., in the FNO case?

The approximation rate of basis expansions is generally determined by the choice of basis $\varphi, \psi$, the properties of the target function (function space), and the norm used to measure the error. There is an extensive body of work on this topic, necessitating a literature review. Here, we briefly outline some well-known results. In the case of Fourier bases on $d$-dimensional torus $\mathbb T^d$, it can be shown by Parseval's identity that the approximation rate in $L^2$-norm is $O(N^{-\sigma + d/2})$ ($\sigma>d/2$) if the target function belongs to the $L^2$ type Sobolev space $H^\sigma (\mathbb T^d)$ (see [Dya1995] for $L^p$-framework). As another example, for many wavelet bases, the same approximation rate $O(N^{-s/d})$ is known based on Jackson-type estimates (see Theorem 4.3.2 in [Coh2003]). Furthermore, there are cases where the rates differ between linear approximation and nonlinear approximation (wavelet approximation); in particular, when the target function is discountinuous (see Remark 4.2.5 in [Coh2003]). We added these discussions after Theorem 1 in the revised version.

References

[Coh2003] Cohen, Albert. Numerical analysis of wavelet methods. Elsevier, 2003.

[Dya1995] Dyachenko, M. I., The rate of u-convergence of multiple Fourier series, Acta Mathematica Hungarica {\bf 68}, no. 1-2 (1995), 55--70.

I would like to thank the authors for the clarifications, these have addressed most of my concerns. Although it is still not fully clear to what extent these results are really applicable (FNO example is for 1-dimensional data, WNO example with Haar wavelets which have not been used in practice), I think the results provide a valuable contribution. I have increased my score accordingly.

We appreciate the detailed suggestions, criticisms and endorsement of the reviewer. We address all of these below:

Minor: typo on line 182 "elliptic operato,r " ... well as I may not have understood everything.

We thanks for finding typo. We modified this in the revised version.

I do not think it is easy to find a Green's function ... how can you do that, if that is the case?]

As you pointed out, the construction or accurate approximation of the Green function is key to realizing the idea proposed in this study. If a good approximation of the Green function is obtained, nonlinear solutions can be constructed using arguments based on Picard's iteration.

There are various approaches to approximating or constructing the Green function depending on the situation. For example, in cases such as $\mathcal L = -\Delta$ on $\mathbb R^d$ or $[0,1]^d$ with the zero Dirichlet or Neumann boundary condition, the explicit representations of the Green functions are known and may suffice. However, in many cases where such explicit representations are unknown, suitable approximations of the Green function are necessary. In this study, we adopted an approach based on basis function approximations. For Fourier bases and wavelet bases, approximation theories are well-established, and using these, the Green function can be approximated (under an appropriate topology) in our setting.

Additionally, approaches that do not rely on basis expansions can also be considered. For example, approximating the Green function using neural networks (e.g., DeepONet) and combining such approximators with the ideas proposed in this study may allow for effective approximation of solution operators for nonlinear PDEs. As described above, various approaches are possible depending on the situation, and we consider these to be valuable directions for future work.

Could you clarify if the error being log $((1/eps)^{-1})$ implies exponential dependency?

In Theorem 1, the necessary depth $L$ is estimated by $L \approx (\log (\epsilon^{-1} ))^2$, which is not exponentially, but squared logarithmically increasing. While, the necessary number $H$ of neurons is estimated by $H \approx \epsilon^{-1}$, which is polynomial increasing. We added this comment after Theorem 1 in the revised version.

How can I interpret $K^{(l)}_N$? ... unless it is specific to the Neural Operator?)

$K^{(l)}_N$ are non-local operators which learns the non-local behavior in PDEs. We can interpret FNOs [Li et al, 2020a] as the special case when $\varphi$ and $\psi$ in $K^{(l)}_N$ are chosen as Fourier basis. We added more explanations of non-local operators $K^{(l)}_N$ after Definition 1 of the revised version.

Can you clarify (or add in the paper to ... (I checked other sources to understand).

The rank of neural operators corresponds to the number $N$ of the truncation of basis expansion in non-local operators $K^{(\ell)}_N$. We already mentioned the rank $N$ in the last line of Definition 1.

It seems the first 2 sentences of ... and is just a first step?

We agree that the first 2 sentences have contradictory. In the revised version, we removed the first sentence "A limitation of our method is that it applies only to parabolic PDEs that can be solved using the Banach’s fixed point theory", which is not a limitation of our works. In addition, we added the sentence ``In this paper we focus on parabolic PDEs with power-type (or somewhat general) nonlinear terms as a first step of study of neural operators based on Picard's iteration." in Section 5 of the revised version.

For the long-time solution, how reasonable is it that ... expressiveness of the solution of the PDE?

This strongly depends on the initial conditions and the nonlinear term. For example, consider the PDEs $\partial_t u - \Delta u = |u|^{p-1} u$. When $1<p \le 1+2/d$ (which is known as the Fujita exponent), it is known that any positive solution blows up in finite time. As a result, $u(T)$ will necessarily exceed $B_{L^\infty}(R)$ at some time. On the other hand, when $p> 1+2/d$, if $\lVert u_0 \rVert_{L^\infty}$ is taken sufficiently small, then the solution $u$ can be extended globally in time. In this case, $u(T)$ always remain within $B_{L^\infty}(R)$. Moreover, there is an extensive body of research on the criteria for determining global solutions and blow-up solutions of PDEs.

We appreciate the detailed suggestions, criticisms and endorsement of the reviewer. We address all of these below:

The assumptions on the operator L and ... the scope of applicability. 4. Can the authors elaborate on how the ... operator and boundary conditions?

In this paper, we focused on parabolic PDEs with power-type (or somewhat general) nonlinear terms
as a first step of study of neural operators based on Picard's iteration. However, we believe that our approach is not restricted to parabolic PDEs and can be adapted to a broader class of nonlinear PDEs, because the Banach’s fixed point theory is widely applicable to many nonlinear PDEs. Below, we provide additional remarks on Assumptions 1 and 2, and some remarks on applicability and limitation, which were added in Appendix G of the revised version.

Regarding Assumption 1: Assumption 1 for $\mathcal{L}$ is a condition satisfied by the solution operators of many linear parabolic PDEs. In particular, it generalizes the rate of time $t$ using the parameter $\nu$, and it is known that many important operators $\mathcal{L}$ satisfy Assumption 1, as stated in Appendix~A.

On the other hand, there exist examples such that Assumption 1 does not hold. For instance, Schrödinger operators with inverse square potentials, i.e., $\mathcal L = -\Delta + \frac{c}{|x|^2}$ (where $c$ is a real number greater than the negative of the best constant of Hardy's inequality), are known to satisfy the inequality in Assumption 1 when $\nu = d/2$ and $d \geq 3$, provided that $\frac{2d}{d+2} \leq q_1 \leq q_2 \leq \frac{2d}{d-2}$ (this range is optimal. See [IMSS2016] and [IO2019]). Thus, due to the constraints on the range of $q_1$ and $q_2$, certain operators may fail to satisfy Assumption 1. By extending the range of parameters in Assumption 1 to $q_{\min} \leq q_1 \leq q_2 \leq q_{\max}$, it is possible to generalize the assumption to include such examples. In this case, $q_{\min}$ and $q_{\max}$, in addition to $\nu$, would influence the conditions for well-posedness and approximation error estimates.

Regarding Assumption 2: There are various examples of nonlinear terms, such as $F(u) = |\nabla u|^p$ or $F(u) = u (e^{|u|^2} - 1)$, that do not satisfy Assumption 2. However, we believe that the arguments in this paper can be extended to such nonlinear terms as well. For example, the heat equation with $F(u) = |\nabla u|^p$, known as the viscous Hamilton-Jacobi equation, has its well-posedness established via a fixed-point argument (see [BMS2002]). Similarly, this is also true for $F(u) = u (e^{|u|^2} - 1)$ or more general exponential-type nonlinear terms (see [IJMS2014]).

The reason why our paper focused on Assumption 2 is because the results on well-posedness vary significantly depending on the nonlinear term and it is difficult to unify these results. Therefore, as a first step in this work, we focused on power-type nonlinear terms. As mentioned in Remark~1, smooth nonlinear terms can often be represented by their leading term in a power-type form through Taylor expansion, highlighting the importance of power-type nonlinear terms from this perspective.

Regarding other equations: We expect that similar arguments can be applied to a broader class of PDEs. For instance, initial(-boundary) value problems for various PDEs, such as the nonlinear Schr"odinger equations and nonlinear wave equations (see [Caz2003], [CH1998]) and the Navier-Stokes equations (see [GM1985]), can be expressed as integral equations. By appropriately choosing the initial data space and the solution space, their well-posedness can be shown by using the fixed-point argument.

On the other hand, for PDEs that involve nonlinearity in the principal part such as $p$-Laplacian equations $\partial_t u -\Delta_p u=F(u)$ with $\Delta_p := \operatorname{div} (|\nabla u|^{p-2} \nabla u)$ and $p\not = 2$, it may not be possible to represent them in the form of integral equations. Such equations might not be effectively handled by our approach.