Improved Operator Learning by Orthogonal Attention

Thank you for your detailed, helpful feedback! We're glad that you found our paper to be well-written and insightful in terms of the theoretical foundation. We address your concerns point by point below.

Q1 : Details and clarifications about the choices of hyperparameters.

The size of $d'$ is always fixed to be equal to $d$ in $h_{i}^{(l)}$ to reduce the choice of hyperparamters in our experiments. $d'$ is typically 8 (for Elasticity, Plasticity, Pipe and Airfoil) or 16 (for Darcy and NS2d), which is much less than the width of the neural operator (128).

Q2: Runtime comparison.

In the revision, we provide the runtime comparison on the Darcy benchmark in the appendix B. The results are shown as below:

Our implementation includes two types of NN blocks: Linear Transformer ("LT") and Nystrom Transformer ("NT"). The results demonstrate that the time cost of our neural operator is acceptable compared to the baselines, and the complexity primarily originates from the NN block. We believe that these findings provide valuable insights into the model complexity. Thanks to your suggestion.

Q3: Applying multi-head attention.

We think it's an insightful suggestion to improve our proposed attention. We assume that the orthogonalization process is applied after the input is transformed into multiple attention heads. Theoretically, it will become kernel integral operator with multiple kernels, which may enrich the ability. We leave this as an interesting direction for future work.

Q4: The position of the activation function.

We noticed that the sentence can be misleading and revised in the revision. We clarify that the activation function is applied after the first linear layer of the two-layer FFN and a simple linear layer without activation function is applied in the last layer.

Q5: The caption of Figure 2.

We notice that Equation 5 can be misleading in the computation order. In practice, we will compute as $\tilde{h}^{(l+1)}_{i}= \hat{\psi}^{(l)}_i\mathrm{diag}(\hat{\boldsymbol{\mu}}^{(l)})[{\hat{\psi}^{(l)}_i}{}^{\top}( h^{(l)}_i w^{(l)}_V)]$ to maintain a linear complexity in our attention module. We have revised both the caption and the equation in the revision. Thanks to your comment!

Thanks to your useful feedback again! Given these encouraging new results, we hope you might be convinced to increase your score. We appreciate your feedback strengthening our paper!

Thank you for your useful suggestions! We appreciate your positive feedback on the clarity and comprehensibility of our method. We address your concerns point by point below.

Q1: Performance versus model sizes.

We have conducted experiments about the width and the number of layers of our model in our paper. Please see section 4.4 for more details. The results show that our neural operator improves as we increase the width or layers. We further increased the layers to 30 and parameters to 10M, leading to notable enhancements on both the Elasticity and Plasticity benchmarks.

Q2: The measurement of the overfitting.

We mainly demonstrate the issue of overfitting by comparing the performance of our proposed neural operator with varying amounts of training data against the baseline Geo-FNO. In the latest revision of our paper, we have incorporated another well-acknowledged neural operator, DeepONet, as an additional baseline for comparison. The results highlight that our model achieves comparable performance while requiring significantly less training data. Additionally, our model exhibits lower performance degradation compared to the baselines. We also experimented with increasing the number of layers to 30 on the Elasticity and Plasticity benchmark. Our neural operator demonstrates significant improvement without exhibiting overfitting. This is in contrast to the performance of Geo-FNO, which shows a decline in performance when the number of layers exceeded 4 in [1]. This observation suggests that our proposed neural operator offers an effective regularization mechanism that contributes to its enhanced performance compared to the baseline.
We believe that these results effectively showcase the problem of overfitting and highlight the regularization benefits of our proposed orthogonal attention.

We sincerely hope for a reconsideration of our paper's score. If you have any more questions or find some mistakes, please feel free to correct us.

[1] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023

Dear reviewer bpG7,

As the discussion session draws to a close, we would like to inquire if there are any additional comments or clarifications you would like to make. We are more than willing to provide responses to any inquiries and address any feedback you may have. Thanks!

Thank you for your helpful suggestions! We are glad that you think the orthogonal attention is "novel" and the two-pathway architecture is innovative. We address your concerns point by point below.

Q1: Empirical analysis of overfitting.

Thanks to your comments about this. We have performed an experiment on the Elasticity benchmark, varying the number of training data, to illustrate the issue of overfitting when data is limited. Our results indicate that our neural operator exhibits a smaller decrease in performance with fewer training data compared to the baseline model Geo-FNO. We have also included another well-acknowledged neural operator, DeepONet, as an additional baseline in our revised version. This addition serves to further highlight the issue and strengthen the motivation behind our work. What's more, we have also experimented with increasing the number of layers to 30 on the Elasticity benchmark. In contrast, Geo-FNO showed a decline in performance when exceeding 4 layers in [1]. This suggests that our proposed neural operator incorporates effective regularization mechanisms, contributing to its superior performance compared to the baseline model.

Q2: Theoretical analysis of orthogonal attention.

Our proposed attention is based on the insight of eigendecomposition. We have also included theoretical support in the form of Eq (17) and Eq (18) in the appendix A, which demonstrates that the parametric $\hat{\psi}$ will converge to the top-$k$ principal eigenfunctions of the unknown ground-truth kernel integral operator. We think it's interesting and inspiring to relate our attention to spectral properties and really appreciate your suggestion.

Q3: The impact of the disentangled design.

Thanks for your interest in the disentangled design. We think that the orthogonal attention naturally requires two pathways and their inputs: the eigenfunctions and the input function, making it challenging to replace the disentangled design.

Q4: Comprehensive evaluation in complex settings.

We agree that more complex practical settings can evaluate the neural operator more comprehensively. We are also interested in the performance in practical scenarios. However, we emphasize that our current empirical studies align with the related works in this field and can prove the effectiveness of our method. We will explore more complex tasks in the next version.

We welcome any questions or corrections you may have and sincerely hope for a reconsideration of our paper's score. Your feedback is highly valuable to us.

[1] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023

Thank you for the valuable feedback! We greatly appreciate your recognition of the novelty of our attention mechanism. We would like to address your concerns point by point as follows.

Q1: Incorporating the orthogonal attention into both lines.

Thanks to this comment. We think that the bottom line requires expressive structures to extract features from the input data and approximate the eigenfunctions rather than structures with regularization that may limit the overall expressiveness of the neural operator. In fact, we conducted such an experiment on the Darcy benchmark and observed that the neural operator exhibits unsatisfactory performance (0.036 versus 0.0072 on Darcy). Hence, we recommend employing a well-studied linear attention instead of orthogonal attention in the bottom line.

Q2: Simplifying the orthogonalization process.

We appreciate your insightful advice regarding simplifying the orthogonalization process. We understand your concern about the complexity involved in the process. The complexity of our proposed process mainly relies on the Cholesky decomposition of the covariance matrix and matrix inversion. We think that both processes maintain an acceptable complexity of O($8^3$) (for Elasticity, Plasticity, Pipe, and Airfoil) and O($16^3$) (for Darcy and NS2d). In our latest revision, we have provided the time cost compared with baselines in appendix B, which demonstrates that the complexity is reasonable. We acknowledge that the process can be further simplified and we are grateful for your suggestion. We will explore this in our future work.

If you have any more questions or find some mistakes, please feel free to correct us. We sincerely hope that you will reconsider and potentially increase the score for our paper.