Optimization for Neural Operator Learning: Wider Networks are Better

1. Although the writing in this paper is very concise and clear, it lacks professionalism, especially in highlighting its technical contributions and innovations. For instance, compared to ordinary Feedforward Neural Networks (FNNs), what are the significant challenges in error estimation for neural operators, and what new techniques are necessary to derive their convergence bound? Moreover, the better performance of wider and deeper neural networks in fitting data is already a well-acknowledged fact in the field, so deriving this conclusion cannot be considered a major contribution of this paper.
2. The paper derives the convergence rate for two neural operators (FNO and DeepONet) but does not provide a comparison between them, nor does it elaborate on their differences and connections. Additionally, there is no discussion on whether the derivation method proposed in this paper can be applied to or how it might be improved for neural operators beyond the aforementioned two structures.
3. The experimental section could benefit from additional validation. For instance, for the 2-dimensional Darcy flow problem and the 2D time-dependent Navier-Stokes (NS) equations, which are commonly used datasets, it is imperative that the authors also conduct experimental validation on them. Furthermore, since the data used for neural operators adhere to PDEs, verifying which datasets satisfy the assumptions made in the paper would also be valuable.
4. ")" is missing in the title of section 3.2.

In Section 3.1, It seems from the notation of branch and trunk net that $\theta_f$ and $\theta_g$ are shared across all branch and trunk nets respectively, or in other words $\theta_f$ and $\theta_g$ are same for all values of $k$. This does not seem to at all be consistent with how DeepONets work - where usually at each $k$ its a different set of parameters - albeit with an overlap. So the very premise of the paper seems suspect!

Also in Section 3.1, the bold facing of $f, g, u$ and $x$ seem to be inconsistent. For eg. $f_k$ is not bold in the first paragraph, but it is bold in the second. This creates sufficient confusion with regards to following the paper.

In Section 4, Theorem 1 is almost identical to Theorem 5.3 in the Banerjee.et.al (2023). It is not made clear in the writing if this theorem is exactly the same or if it is different, then how exactly does it differ from the other.

In Section 5, Definition 2, $f_{k}^{(i)}$ and $g_{k,j}^{(i)}$ are not defined anywhere.

The $t-$dependence of the potential contraction factor, $\frac{\alpha_t \omega_t (1-\gamma_t)}{\beta} (2 - \omega_t)$, in equation $9$ raises questions about why this should result in geometric convergence. Infact, it is not even transparent that equation 9 is a contraction.

The theoretical results admit interesting conclusions, such as wider networks perform better, but it is unclear to what extent this is specific for neural operators since similar results exist for feedforward networks.

The main concern of the reviewer is novelty, the analysis in the paper is no different from recent optimization analysis of neural networks. The author also still conjectures that wider is better can be an overclaim.

• Why dealing with operator networks is different from neural networks?
• From my understanding, the initialization makes the networks lie in the lazy regime. Is this an interesting regime for analyzing? What if I change the initialization scheme, and the wider the better still holds?
• The lazy regime is not learning a feature, is there a possibility that a finite/smaller width operator network can enforce feature learning.
• Is the wider the better an overclaim? For this paper don't present any result that the narrow network is provably hard (ie lower bound)

Most important, the reviewer can't find any difference between the ananlysis of operator learning and neural network and the conclusion differs from each other. The reviewer highly questions the novelty of the paper.