Understanding Optimization of Operator Networks with Variational Loss for Solving PDEs

• There is some confusion in the introduction between physics-informed neural networks (for solving known PDEs by minimizing the residual), and neural operator (for learning solution operators without PDE knowledge)
• The paper is not very well structured, and the relevant parts are not very well explained whereas space is wasted on formally introducing neural networks
• The paper is not self-contained and very difficult to understand without reading the paper of Banerjee et al. 2022
• There is some concern about the novelty for the results concerning the condition number of A missing. It is well-known in numerical PDE solvers that smaller condition number improves convergence and preconditioning helps to get smaller condition number
• The main results on convergence rate in Thm 3.4 are not very useful as the convergence rate $r_t$ depends on the quantity $q_t$, which itself depends on the gradient of loss function w.r.t parameters. Hence, one cannot really characterize this as a convergence result given that we have no control on $r_t$.

This paper is extremely difficuly to read, written sloppily, and appears to be significantly underdeveloped. The theory section appears to provide almost no new insights, while the numerics are lacking in both quality and quantity. I expand below:

The effect on the optimization process of the condition number of a linear system for minimizing a linear regression task has been well-known and does not provide any new insights. The proposed adaptive weight algorithm can be interpreted simply as time-dependent pre-conditioning. However, once such a time-dependent pre-conditioning is introduced, theorem 3.4 no longer holds as the algorithm is no longer that of gradient descent, and a new result is required for the updated algorithm - however such an adaptation is not present in the paper.

The numerical section appears to cover only one (there appears to be another one in the appendix, which paints a different picture - showing that adaptive weighting does almost no better than other preconditioning strategies) experiment, reporting only a single relative L2 error (is this over a test set or a single solution?) - making it extremely unclear whether the results are generalizable. The table trial errors do not appear to agree with the graphs provided, however the training curves also do not appear to agree with the original ULGNet of Choi et al. 2023. What is more - the Trial A appears to begin at a much larger loss value, raising the question of whether the loss values reported always correspond to the same loss or different losses - i.e. is it $L^M$ for all runs or $\hat{L}^M$ for some?

I further highlight other issues in detail in the questions section.

Regarding the theoretical results, I have serious concerns about two specific areas that have not been proved and which imply the proofs are incomplete and therefore not publishable yet.

• In Lemma 3.2, it has not been proved that the set $Q_q\\cap B^{Spec}_{\\rho,\\rho_1}\cap B^{Euc}_{\\rho_2}$ is non-empty. I haven’t found the proof for this in the Appendix. If this can’t be proved, then there is no existence of $\\theta’$ and so the equation in Lemma 3.2 is void of meaning, as well as the optimization guarantees in Theorem 3.4 that uses this result—i.e., the main contribution of the paper. The proof is needed urgently for the main theoretical results to make sense.
• There is a big issue in Theorem 3.4. When looking at its proof, it is important to prove that $\\delta_t$ is less than $1$. However, in line 954, when trying to prove this, the authors simply claim that because of the “definition of $Q_{q_t}(\\theta_t)$”, we have that “$\\theta_{t+1}\\in Q_{q_t}(\\theta_t)$”. This claim is not evident and requires a proof—in my opinion, it is a baseless claim. The RSC set $Q_{q_t}(\\theta_t)$ proposed by the authors is different than the RSC set established in Banerjee et al., 2022 (the work that the authors based their results on), and so the results for the RSC set in Banerjee et al., 2022 cannot be applied “immediately” to solve this problem. A proof is needed.

There is a lot of notation problems that may have strong repercussions in the understanding of the paper and possible of the validity of its results. It also shows the paper does not seem to have been proofread. As it is, the paper is not in a suitable state for publication, mainly for the issues I have found below:

• Line 100 mentioned that $M$ is the dimension of the input feature (or data) $\\omega$; however, in equation (6) and later in the same paper $M$ denotes the sample size! This is confusing. Moreover, in Assumption 3, $M$ is used again as saying that it is a constant that simply “exists” for every $\\omega$.
• Another big problem is the use of the symbol $\\omega$. In line 100 it is defined to be an “input feature” which seems to be just data, since it is used in the equation of a neural network. The problem is that then around line 152, it is said to be a member of the “parameter space”, which is a confusing term given that before the paper related $\\omega$ with data. Could the authors make more precise what $\\omega$ represents? I also want to suggest explicitly stating that $\\omega$ is a vector.
• Many typos: for example, delete “through” in line 031; in line 108 it should say “of all layers”; the “(\mathbf{x})” missing in equation (1); the dimension of $\\bar{\\theta}$ in line 234 should be $p$ instead of $m$; the subscript $2$ must be added to the norm notation in Definition 3.1; remove the $M$ in the denominator from the fraction in lines 1019-1024. The authors should afterwards proofread the whole paper and correct all writing errors.

Finally, another considerable issue is that the simulations are missing a stronger connection with the guarantees established in Theorem 3.4 and the RSC framework. Theorem 3.4 shows that the convergence rate depends $r_t$, which depends on a different between $q_t$ and a quantity that has $m$ in its denominator. Thus, there seems to be a dependency of the convergence rate on the neural network’s width $m$. Simulations should be done for every of the five trial settings described in Section 5 where now the width values $m$ are changed. The authors should then comment on any observed change in the convergence rate. This would make the simulations explore further the implications of the theoretical derivations.

Other issues:

• The symbol $\\mathbf{A}$ and $q_t$ is introduced in both the abstract and Introduction without any explanation of what they really are. I would recommend to either remove them or give a more anticipatory explanation of what they are.
• Add the Banerjee et al., 2022 citation to the end of the first sentence in line 071.
• Update the Banerjee et al., 2022 citation since I found a published version in ICLR 2023.
• In the contributions, both “training efficiency” and “training performance” are mentioned: what are the differences between those two terms? They must be explained.
• In Section 2.1, specify that $\\phi$ is applied entry-wise when it has a vector as an argument.
• What does “uniformly elliptic” mean in line 121?