We thank Reviewer kDgT for the detailed feedback. Below, we respond to the questions.

W1: Experimental repetition.

The results for neural operators are relatively stable. Consequently, many results for widely used baselines are taken directly from the original papers, which also do not include repeated experiments [1,2]. Reproducing experiments for every baseline and benchmark multiple times is computationally intensive. To address this concern, we have conducted additional experiments with three repetitions for our method on NS-2D and CFD-2D benchmarks as shown in the table below. The standard deviations are minimal compared to the average values. We will include these results, along with standard deviation values, in the revised manuscript to better assess the statistical stability of our approach. We appreciate your feedback and will incorporate these improvements into the revised version.

W2: Validation dataset.

The primary objective of our experiments is to evaluate the effectiveness of our models across various benchmarks and to provide a comparative analysis with existing baselines. To ensure fairness and consistency, we use a fixed set of training settings in line with other baseline studies [1,2,3]. We do not perform hyperparameter tuning specifically for individual benchmarks in Table 2, as detailed in Appendix B. This approach aligns with standard practices in operator learning and helps mitigate the risk of overfitting by preventing excessive optimization for any single benchmark.

W3: Related works.

We have briefly discussed our choice of neural operators over other neural network-based methods in the introduction. Due to space constraints, a more detailed comparison with other ML models for PDEs will be included in the revised version, potentially in the appendix.

W4: Undefined symbols.

Thank you for pointing this out. In Section 3, $d_a$ and $d_u$ represent the dimensions of different functions, and FFN refers to the feed-forward neural network. We will provide detailed definitions for these terms in the revised version.

W5: More description of the factorization trick.

The "Factorization trick for high-dimensional PDEs" was briefly introduced as it serves primarily as a supplementary technique rather than a core component of our method. We will provide a more detailed mathematical description and explanation in the appendix.

W6: Training/inference time and memory consumption.

We present a comparison of training/inference times and memory consumption for AM-FNO and FNO on the 2D Darcy benchmark with a resolution of $421 \times 421$ in the table below. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. During inference, AM-FNOs (with kernels generated by MLP or KAN being precomputed) exhibit a similar speed to FNO, as both methods rely on similar kernel calculations. We will provide a more detailed discussion of these aspects in the revised version of the paper.

Q1: $a_i$ refers to different input functions indexed by $i$, while $u_i$ denotes the corresponding output functions.

Q2: Thanks for your suggestion. We will describe them differently in the revised version.

Q3: We apologize for the confusion. The caption states, "A version of the model without orthogonal embedding (Non) is included for comparison," which refers to the model that directly approximates the kernel with MLP, as opposed to the KAN approach.

Q4: Citation of "Stone-Weierstrass theorem".

The Stone-Weierstrass theorem specifically addresses polynomial approximation of continuous functions, while our theorem generalizes approximation using orthogonal function bases [4], not limited to orthogonal polynomials. We will add the appropriate reference and clarify this distinction in the revised manuscript.

Q5: Moving the KAN part.

Thank you for your suggestion. While AM-FNO(KAN) does not outperform AM-FNO(MLP), it offers significant advantages, such as extendability during training and interpretability, as discussed in Appendix E. These benefits highlight the value of the KAN approach, which we believe justifies its inclusion in the main text.

We hope this response clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know. We sincerely hope that you will reconsider and potentially increase the score for our paper.

[1]Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.

[2]Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2023). Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388), 1-26.

[3]Li, Z., Meidani, K., & Farimani, A. B. (2022). Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13671.

[4]https://www.math.uni-hamburg.de/home/gunesch/calc1/chapter11.pdf (Theorem 11.13)

Thanks for your response again.

W2:

Thank you for your explanation. We reported the test score evaluated at the last epoch for all the models.

W3:

In our revision, we assume that $\mathcal{F}$ is a separable Hilbert space. According to Theorem 9 in [1], there exists a complete orthonormal system (orthogonal basis), which can approximate any arbitrary function $f$ within the space, as stated in Definition 11.9 and Theorem 11.13 in [2]. We will consider introducing it directly within the text in the revised version. Thank you for your suggestion.

If you have any further questions, feel free to let us know.

[1] https://www.math.nagoya-u.ac.jp/~richard/teaching/s2023/SML_Tue_Tai_2.pdf

[2] https://www.math.uni-hamburg.de/home/gunesch/calc1/chapter11.pdf

Thank you for your reply.

W2:

Please report a part of the revised Table 2 (performance at the final epoch, multi-dimensional PDEs are preferable), to assess the effectiveness of Amortized FNO in comparison to the other models again. I expect that it does not need so much effort but only checking log files...

We thank Reviewer Z566 for the valuable feedback. Below, we respond to the questions. Please note that the additional tables and references are included in the Global Rebuttal due to character limits.

W1: More baselines.

Our method aims to enhance Fourier neural operators (FNOs), a significant subclass of neural operators. Therefore, we primarily use FNOs as our baselines, and our method demonstrates superior performance. Additionally, we have included widely used baselines (OFormer) and a competitive baseline (LSM) to validate our method's effectiveness. In response to your suggestion, we include some mentioned baselines, as shown in Table B. It indicates that our method outperforms the additional baselines as well. We will include these results in the revised version.

W2: Benchmarks on unstructured mesh.

Thanks for your suggestion. We evaluate our method on the Elasticity benchmark presented on point clouds [1] in Table C. To handle irregular geometry, we implemented the widely used Geo-FNO method to map the irregular meshes to and from uniform meshes [1]. Our results show that our method outperforms other neural operators that utilize the same learnable mapping. However, it performs worse than transformer-based neural operators, which naturally process input functions on irregular geometry as a sequence. We hypothesize that the learnable mapping introduces errors. We will include these results in the revised version.

W3: Spectral Neural Operator Resemblance.

We would like to clarify that our method is fundamentally different from SNO. SNO utilizes Chebyshev polynomials to represent functions with finite sets of coefficients and learns the mapping between these coefficients. This results in band-limited operators that are restricted to generating frequencies within a fixed representation space. In contrast, FNOs learn the direct mapping between functions and compute the integral operator in Fourier space for high inference speed. Empirically, FNOs demonstrate the ability to extrapolate to higher frequencies not seen during training, which SNOs lack due to their fixed representation limits [2]. As a variant of FNO, while AM-FNO employs orthogonal basis functions to approximate the kernel function—a common practice in function approximation—the neural operator itself is fundamentally distinct from SNO.

W4: Low-rank approximation or depthwise convolution with non-linearity.

FNO with low-rank approximation is similar to one of our baselines, F-FNO, which factorizes the kernel across different dimensions. The results in Table 2 (in the paper) demonstrate that AM-FNO outperforms F-FNO, highlighting the superiority of our amortized parameterization over the point-by-point parameterization in F-FNO.

Regarding the non-linear convolution, AFNO [3] employs a non-linear convnet to evolve features at every frequency mode. We evaluate AFNO on two benchmarks, and the results, as shown in Table B, indicate that AFNO has lower accuracy. We hypothesize that this is because AFNO uses the same MLP to evolve all frequency modes of the input, potentially limiting expressiveness. In contrast, AM-FNO transforms different frequency modes separately. We will include further discussion on AFNO in the revised version.

Q1: Our rationale for employing orthogonal basis functions stems from the observed performance degradation when using an MLP to directly approximate the mapping between frequencies and kernel values, as shown in Table 4 (in the paper). The MLP struggles to capture the complex, non-linear nature of the kernel function. Embedding scalar inputs with predefined functions is a well-established technique in machine learning, as exemplified by time embeddings in diffusion models. This approach helps MLPs convert linear inputs into expressive, non-linear forms effectively. Inspired by it, we employ orthogonal functions to embed the frequencies, mapping them into a high-dimensional and non-linear feature space. This introduces an inductive bias that enhances the efficiency of approximating kernel functions. Regarding the depthwise convolution in FFT space, please refer to our response in W4.

Q2: As reported in [4], KAN does not consistently outperform MLP. As shown in Table 4 (in the paper), AM-FNO with KAN outperforms AM-FNO using MLP directly and performs worse than AM-FNO using orthogonal basis functions to embed the input. This indicates the effectiveness of our embedding technique. Meanwhile, KAN has the advantage of being extendable during training by increasing the number of local basis functions. For a fair comparison, we kept the parameter count constant and did not extend KAN during training. Given the results in Figure 4 (in the paper), it is plausible that AM-FNO (KAN) can outperform if we increase the number of basis functions during training.

Q3: The F-FNO results were reproduced using the training settings and similar hyperparameters as other FNOs, as described in Section 5.1 and Appendix B, to ensure a fair comparison. The discrepancy with the original paper's results may stem from differences in training techniques. For instance, the original F-FNO paper employs methods like enforcing the first-order Markov property and adding Gaussian noise. Our training settings align with standard practices in this domain to maintain consistency across evaluations.

Q4: Please refer to W4.

Q5: Please refer to W3.

Q6: Please refer to Q1 and Q2. We primarily used Chebyshev basis functions for the orthogonal embeddings in our experiments. In our ablation study, we also replaced them with triangular basis functions and non-orthogonal basis functions to evaluate their impact.

We hope this response clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know. We sincerely hope that you will reconsider and potentially increase the score for our paper.

Thank you for the response.

1. The paper references factorization techniques, but the implementation uses a Chebyshev basis to parametrize the entire Fourier kernel, along with an MLP. As reviewer DLAH noted, this approach is not clearly explained in the current version of the paper, where only smooth function parametrization is mentioned. It is essential to explicitly include this discussion in the paper for clarity and the discussion about the spectral bias in MLP.

2. I couldn't find hyperparameter details for the benchmark datasets in the paper, which is needed to ensure reproducibility.

3. If you are using all Fourier frequency modes in the proposed method, then it's not a fair comparison. I would like to see the performance compared with FNO, where we have used all the frequency modes. Also, it would be great if the author could provide the performance of the proposed method using only the same number of modes as used for FNO.

4. I could find two Tx and Ty in the code. Could you clarify:

self.Tx = torch.zeros(self.n1, H+padding)

    self.Ty = torch.zeros(self.n2, (W+padding)//2+1)

    self.Tx = (torch.cos(self.grade1@torch.acos(self.gridx))).reshape(1, self.n1, H+padding, 1).cuda() 
    self.Ty = (torch.cos(self.grade2@torch.acos(self.gridy))).reshape(1, self.n2, 1, (W+padding)//2+1).cuda() 

5) Could you compare the proposed method with baselines regarding training time, GPU consumption, inference time, and number of parameters?

PS: I am open to discussing the paper and would like to consider increasing the score if my questions are addressed.

Thanks for your feedback.

1. Thanks for your suggestion. We agree that the concept of spectral bias provides valuable insight into our approach, and we will include additional discussion to elaborate on this in the revised version.

2. Sorry for not including this part. We will include the relevant descriptions of the benchmarks in the revised version.

3. In our main experiment (Table 2), we aimed to ensure that all models had a comparable number of parameters. However, the parameter count for FNO with all modes becomes excessively large, making it unfair to compare with other models. Meanwhile, one advantage of our method is that it can capture all modes with a limited number of parameters. We have presented the results of FNO with full modes on CFD-2D (denoted as "FNO+" in Table 3), where AM-FNOs outperform FNO+. We also provide comparisons with FNO+ on the Darcy and Airfoil benchmarks below. We will include additional empirical results on FNO+ and AM-FNOs with the same number of modes in the revised version.

4. The first two lines of code are for initialization (though they aren’t necessary), while the next two lines compute the basis functions. We want to clarify that our code is consistent with our method.

5. We provide a comparison of all models, using the hyperparameters from the main experiment, tested on the same benchmark in the table below. Our methods require more memory and training/inference time compared to other FNOs, due to the additional modes we maintain. However, AM-FNOs still outperform non-FNO neural operators.

If you have any further questions, we are pleased to discuss them.

Thank you for the response.

Q2) It would be better if you could also report hyperparameters for the proposed method.

Q3) With increased modes, FNO is computationally heavy and might sometimes underperform. However, can you report the numbers of AM-FNO compared with FNO having the same modes? It will help to understand the proposed method better, whether increasing the modes is effective in AM-FNO or the way in which the kernel is parametrized (Kernel Bias)

Q) Can you report the numbers using just one layer of AM-FNO and FNO on specific benchmarks?

I am trying to understand the proposed method better from the point where kernel bias is more critical or increasing the modes. Also, it's fine if the proposed method is not competitive with FNO in terms of computational and time complexity, as the proposed method is trying to address different problems altogether.

Thanks for your feedback.

Q2) We have reported the essential hyperparameters of our models in Section 5.1. We will provide a more detailed description in our revised version.

Q3) We present the performance of our models with 12 modes compared to FNO with the same number of modes on Darcy benchmark, as shown in the table below. The results indicate that our models continue to significantly outperform the FNO, demonstrating the effectiveness of our parameterization.

Q) The results of AM-FNOs and FNO with one layer on Darcy are shown below.

If you have any further questions, we are pleased to discuss them.

We thank Reviewer NuVs for the valuable feedback. Below, we respond to the questions.

W1: The reason for tackling the CoD issue in the number of parameters.

In FNO, the kernel is parameterized independently for each frequency mode, resulting in complexity for the Fourier integral operator of $O((d_h^2 k)^{D}))$, where $d_h$ represents both the input and output channel dimensions, $k$ denotes the number of retained modes, and D indicates the spatial dimensionality. For high-resolution and high-dimensional data, this leads to significant memory consumption, as a large number of modes need to be retained. For example, one 64-width FNO layer for 2D PDEs can exceed 130 million parameters with $k=256$. This severely limits practical applications, such as large-scale pre-trained models.

We provide a comparison of the training time and memory consumption for AM-FNOs and FNO ( with all modes retained) in the table below, tested on the 2D Darcy benchmark with a resolution of $421 \times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions, attributed to AM-FNO's reduced complexity. A comprehensive analysis will be included in the updated version of the paper.

W2: Paper presentation.

Thank you for your feedback on the presentation. We apologize for any confusion caused by the brief introduction of FNO due to page limitations. We will provide a clearer explanation in the updated version to ensure understanding, especially for readers not familiar with the implementation details.

Q1: Parameter reduction

We would like to clarify that in AM-FNO, the MLP is used to map each frequency mode (after embedding) to its corresponding kernel value, meaning the number of parameters depends on the number of channels rather than the number of retained modes. For instance, if an MLP with $m$ basis functions processes input and outputs a kernel of size $k$, the parameter count is approximately $2m^2 + 2m$ (assuming one hidden layer and one channel). In contrast, FNO requires $k$ parameters for the kernel, which can be expensive for high-resolution and high-dimensional data (more modes should be retained).

Q2: MLP limits the expressivity.

While Fourier methods are theoretically efficient, they require a large number of parameters when handling high-dimensional and high-resolution data, which can be computationally prohibitive. Using an MLP to generate the kernel might indeed limit expressivity compared to parameterizing every frequency mode as in FNO theoretically. We regard this as a tradeoff. However, our empirical results demonstrate that AM-FNO outperforms FNO covering all frequency modes (see Table 3). We believe this is because the smoother transformations facilitated by the MLP improve optimization efficiency, leading to better performance.

Q3: Main experiments.

Many results in Table 2 are sourced from the original papers, making a direct comparison of running times potentially unfair due to hardware differences. All the baselines in Table 2 have comparable numbers of parameters, with their hyperparameters detailed in Appendix B. Below, we provide a parameter comparison on Darcy benchmark to validate this. The results of neural operators are generally consistent despite randomness, which is why many popular baselines do not report repeated results [1,2]. We repeated our experiments three times on CFD-2D and NS-2D benchmarks, as shown in the table below, demonstrating the stability of our method.

Thank you for your openness and willingness to discuss further. We hope this discussion clarifies any misunderstandings and addresses your concerns. If you have any further questions or identify any mistakes, please do not hesitate to let us know.

[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.

[2] Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., ... & Zhu, J. (2023, July). Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning (pp. 12556-12569). PMLR.

Thanks for your prompt and detailed response!

W1:

We appreciate your feedback and would like to address your concerns subsequently.

Firstly, we apologize for the error in the complexity statement; it should be $O(d_h^2k^{D})$.

Secondly, there have been attempts to utilize large pre-trained FNO models. For instance, DPOT [1] explores a model that employs a shared MLP to transform each frequency mode of the input (akin to a convnet with a $1 \times 1$ kernel) to mitigate the memory overhead of kernels. This architecture is similar to AFNO [2], which we have compared in Table B of the global rebuttal. Our results indicate that our method outperforms AFNO on both benchmarks, which we attribute to AFNO’s limited expressiveness due to its uniform transformation across different frequency modes, while our approach treats each mode differently.

Regarding your concerns about FFT, we acknowledge that FFT suffers from the Curse of Dimensionality (CoD). However, FNOs still demonstrate superior training speed (as shown in the table below in Q3) and prediction accuracy compared to other neural operators. Our method primarily addresses memory issues, particularly in high-resolution or high-dimensional contexts, as demonstrated in the table in the rebuttal before. Our approach shows a reduction in memory usage (and shorter training time for AM-FNO (MLP)) even in 2D benchmarks, suggesting potential benefits for large-scale models.

Q1:

Continuing with the assumption of one channel and a one-dimensional kernel size of $k$, let’s further simplify the MLP to a linear layer for clarity. To map frequency modes (with shape $[k,1]$ ) to the corresponding kernel values (also with shape $[k,1]$ ), our method first applies basis functions to obtain an embedding with shape $[k,m]$ , where m is the number of basis functions. We then compute the kernel values (real or imaginary part) using a linear layer with $m \times 1$ parameters. Consequently, the complexity of our method depends only on the number of channels and the number of basis functions (specifically $O(D m d_h^2)$ in the linear case compared to $O(d_h^2k^{D})$ in FNO for multi-dimensional PDEs), and avoid becoming excessively large with increasing dimensions and resolution.

Q3:

To address your concern, we provide a comparison of training times under our experimental settings for the same benchmark, as shown in the table below. The results indicate that our method has a slower training speed compared to other FNOs, which can be attributed to the additional frequency modes we adopted. However, our method still outperforms non-FNO neural operators. Furthermore, as shown in the previous rebuttal, when all modes are retained, our method can surpass FNO in training time.

If you have any further questions, we are pleased to discuss them.

[1] Hao, Z., Su, C., Liu, S., Berner, J., Ying, C., Su, H., ... & Zhu, J. (2024). Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03542.

[2] Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021). Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587.

Thank you for making such a constructive discussion with us!

We totally understand your concerns about method efficiency and have tried our best to demonstrate that our approach may not suffer from issues from that aspect during the rebuttal. Of course, we will add the mentioned comparison between large FNO and large AM-FNO in the revision. Additionally, we will provide a comprehensive comparison with DeepONet, particularly focusing on efficiency.

At last, we also clarify that alleviating the parameter count issue is a side product of this paper. A more evident effect of the proposed amortized parameterization is to foster the frequency modes to communicate with each other, leading to enhanced predictive performance. We will carefully revise the paper to weaken the argument on addressing COD in our revised version.

Thanks again!

Thanks for your feedback and improved score. Below, we further respond to your questions.

1） Due to the time constraints of Rebuttal, we directly used the official Geo-FNO code to compare the speed of Geo-FNO and DeepONet [1]. Note that DeepONet requires function values at all input points and a query coordinate but outputs only the value at the coordinate, while FNO outputs values for all points. To align batch sizes, we used a batch size of $16$ for Geo-FNO and $16 \times 972$ (972 is the number of sensor points) for DeepONet. As shown in the table below, huge batch sizes for DeepONet can pose significant challenges for data transfer and parallel processing, resulting in reduced efficiency. While FFT may become a limitation in very high-dimensional settings, FNO still outperforms in this context. We also compare FLOPs for computing an entire function (972 points) to better illustrate the efficiency of the two models.

Regarding U-Net, while it may be more efficient in many scenarios, its local convolution in the spatial domain limits performance when testing across different discretizations, which is significant for neural operators [2].

Thank you for your questions on efficiency. We recognize that the complexity of FFT may pose a limitation for FNOs and will include a detailed discussion on COD in our revised version.

2）The Plasticity benchmark is presented on a structured mesh, where the indexing induces a canonical coordinate map and enables the direct application of FFT without the learnable mapping [3]. Consequently, Geo-FNO is equivalent to the standard FNO in this context. We refer to it as Geo-FNO to maintain consistency with the original Geo-FNO paper. Thank you for pointing this out.

We hope the above response can further address your concern. If you have any further questions, we are pleased to discuss them.

[1] https://github.com/neuraloperator/Geo-FNO

[2] Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., & Benson, S. M. (2022). U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163, 104180.

[3] Li, Z., Huang, D. Z., Liu, B., & Anandkumar, A. (2023). Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388), 1-26.

We sincerely thank Reviewer okt7 for recognizing the novelty and empirical contributions of our method. Below, we address the questions raised.

W1: Inadequate baseline tuning.

We appreciate the reviewer's concern regarding baseline tuning. To ensure a fair comparison, we have adjusted all models in Table 2 to maintain comparable parameters, with hyperparameters detailed in Appendix B. Specifically, we standardized the width and depth across all FNOs baselines (including AM-FNO) to assess the effectiveness of our amortized parameterization. Furthermore, for the critical hyperparameter of FNO, the number of retained modes, we increased it to cover all modes as demonstrated in Table 3. Even under these conditions, our methods consistently outperformed the baselines.

W2: Limited discussion on scalability.

Currently, our study focuses on 1D and 2D PDEs. As demonstrated in Table 3, covering all frequency modes with FNO requires nearly 9 times the number of parameters, which highlights its limitations even in 2D PDEs. We evaluate our method on a 3D benchmark, Plasticity [1], and the results, as shown in the table below, illustrate its efficiency and scalability. We will provide a more detailed discussion in the revised version.

W3:Computational efficiency.

In the table below, we compare the training times, inference speeds, and GPU memory usage for AM-FNOs and FNO on the 2D Darcy benchmark with a resolution of $421 \times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. During inference, AM-FNOs (with kernels generated by MLP or KAN being precomputed) exhibit a similar speed to FNO, as both methods rely on similar kernel calculations. We will include a more thorough discussion in the revised version of the paper.

Q1 & Q2: Multi-scale, inverse problem and parameter estimation tasks.

We have evaluated AM-FNOs on the Airfoil benchmark, which has some multi-scale features, and achieved state-of-the-art results. A more detailed assessment of AM-FNO on benchmarks with significant multi-scale behavior will be addressed in future work. Our focus is on the forward problem for PDEs. Its effectiveness on inverse problems and parameter estimation is yet to be explored and will be also considered in future research.

Q3: Computational complexity.

Please refer to W3.

Q4: The interpretability of KAN.

Thank you for your insightful question. In AM-FNO, the black-box nature of the MLP between Fourier integral operators limits interpretability. However, leveraging KANs for symbolic expression recovery is promising and could enhance interpretability in future work. This is an area we plan to explore further.

We sincerely appreciate your valuable insights and corrections. We will revise our manuscript accordingly. If you have any further questions or identify any mistakes, please feel free to correct us.

[1] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209,2022.

We thank Reviewer DLAH for the acknowledgment of our method and empirical contributions. Below, we respond to the questions.

W1: The motivation for orthogonal basis functions.

Sorry for the lack of clarity. We make the following clarifications. As shown in Table 4, the performance degrades compared to the version with KAN when using an MLP to directly approximate the mapping between frequencies and kernel values. We attribute this degradation to the MLP's difficulty in capturing the complex, non-linear nature of the kernel function. Embedding scalar inputs with predefined functions is a well-established technique in machine learning, as exemplified by time embeddings in diffusion models. This approach allows MLPs to more effectively transform linear inputs into expressive, non-linear representations. Inspired by this, we employ orthogonal functions to embed the frequencies, mapping them into a high-dimensional and non-linear feature space. This method introduces an inductive bias that enhances the efficiency of approximating kernel functions. This method is supported by the theory of orthogonal functions and empirical results and has been studied in function approximation [1]. Further discussion will be provided in the revised version. Thanks for your question.

W2: Discussion about the Fourier series.

We experimented with triangular basis functions (TBF), which are used as basis functions in the Fourier series, as shown in Table 4. While TBF yielded better results for the Darcy benchmark, they performed worse compared to Chebyshev basis functions in other cases. We hypothesize that TBFs better capture the periodic structure of the Darcy benchmark and will discuss this further in the revised version.

W3: Discussion about AFNO.

Thanks for your suggestion. We evaluated AFNO on two benchmarks, as shown in the table below. The results indicate that AFNO has lower prediction accuracy compared to AM-FNO. We hypothesize that AFNO's lower accuracy is due to using the same MLP to evolve all frequency modes of the input functions, potentially limiting expressiveness, whereas AM-FNO transforms different frequency modes separately. Further discussion on AFNO will be included in the revised version.

Q1: The smoothness about MLP.

The term "more smoothly" in our context indicates that our method achieves smoother evolution compared to FNO, which parameterizes the kernel value point-by-point. This pointwise parameterization may ignore correlations between frequency modes, leading to a less smooth kernel. In contrast, our method uses an MLP to approximate the mapping between frequency modes and their corresponding kernel values, resulting in a smoother representation. As mentioned in W1, directly using an MLP might struggle to capture the complex structure of the kernel, but the orthogonal embedding helps address this issue. However, we believe the improved performance is due to the increased expressiveness to model kernel functions, rather than just a matter of smoothness.

Q2: The central symmetry in Fourier space

Yes. We ensure this central symmetry by obtaining the value at (-k_x, -k_y) from the NN prediction at (k_x, k_y) directly.

Q3: GPU memory and training time comparison.

We provide a comparison of the training time and memory consumption for AM-FNO and FNO (with all modes retained) in the table below, tested on the 2D Darcy benchmark with a resolution of $421 \times 421$. The results indicate that AM-FNO (MLP) exhibits both reduced memory usage and shorter training times compared to FNO, which is attributable to its lower complexity. Although AM-FNO (KAN) demonstrates increased training time due to its architectural design, it still benefits from lower memory consumption. We conjecture that this advantage is particularly evident when solving PDEs with high resolution and high dimensions. A comprehensive analysis will be included in the updated version of the paper.

We are sincerely grateful for your valuable insights, which we firmly believe will significantly enhance the quality of our manuscript. If you have any more questions or find some mistakes, please feel free to correct us.

[1] S Qian, YC Lee, RD Jones, CW Barnes, and K Lee. Function approximation with an orthogonal basis net. In 1990 IJCNN International Joint Conference on Neural Networks, pages 605–619. IEEE, 1990.

Thank you for your response. I understand that the rebuttal period is only one week, and it's not feasible to run many experiments or make significant revisions to the paper. Given the limited time for discussion, I'll focus on my major concern: GPU memory and training time.

Using the full frequency modes (421) for FNO in your comparison of GPU memory and training time is problematic. Previous studies often use a truncation mode of around 12 for FNO, for three main reasons: (1) Keeping too many modes increases computational costs considerably. (2) For PDE data such as Darcy flow, with minimal high-frequency components, using so many modes (421) is redundant. (3) Using too many modes can actually damage FNO's performance.

Therefore, using FNO with a truncation mode of 421 is a very weak baseline in terms of accuracy, GPU memory usage, and time consumption. The current comparison appears to increase FNO's parameters to an unreasonably large number, then claims that AM-FNO uses fewer parameters and GPU memory than this excessively large FNO.

Compared to the commonly used FNO, AM-FNO's architecture is expected to demand more GPU memory and longer training time, which could significantly limit its application for high-dimensional PDE data. Additionally, AM-FNO cannot guarantee it has fewer parameters than the commonly used FNO.

I recommend that the authors provide a more detailed discussion on the efficiency of AM-FNO to avoid presenting potentially misleading results.

We clarify that AM-FNOs use 85×43 modes, whereas FNO uses 2×12×12 modes. AM-FNO requires MLPs to generate the kernel and perform matrix-vector multiplication with a larger kernel. However, both operations can be parallelized for high efficiency.

We present the performance of AM-FNOs trained with $24 \times 12$ modes (requested by Reviewer Z566) on Darcy benchmark, as shown in the table below. Even with truncated modes, AM-FNOs still significantly outperform FNO, demonstrating the effectiveness of MLP parameterization. While maintaining all modes can further improve performance, the impact is relatively modest on this benchmark.

If you have any further questions, we are pleased to discuss them.