FAN: Fourier Analysis Networks

Why hasn't the issue with periodic functions been more widely studied? Are there prior works along these lines? (I don’t completely buy that models like the Transformer can’t model purely periodic functions. Is this a known problem in the literature?...)

The issue of models like transformer having potential flaws in periodicity modeling has already been discussed. Some works also pointed to this argument, which were cited in our Related Work Section. For examples, Works [3] and [4] pointed out that Transformers still struggle with modeling periodicity. Work [5] confirmed that deep neural networks fail to learn purely periodic functions, as demonstrated in its Figure 1, and introduced the Snake function, i.e., $x + \sin^2(x)$, as the activation function to address this issue. However, we observed that it can fit periodic functions to a certain extent, but its effect is limited especially for OOD scenarios, as demonstrated in Appendix D. Therefore, although some previous studies have discussed this issue, their actual performance and range of applications remain heavily constrained.

In this paper, we focus on addressing the issues that existing general neural networks struggle to model the periodicity from data, especially for OOD scenarios. Our proposed FAN performs exceptionally well on periodicity modeling and achieves competitive performance on a range of general real-world tasks.

Reference:
[3] Conditional Generation of Periodic Signals with Fourier-Based Decoder. NeurIPS
[4] Bridging Self-Attention and Time Series Decomposition for Periodic Forecasting. CIKM
[5] Neural Networks Fail to Learn Periodic Functions and How to Fix It. NeurIPS

Why do the FAN Transformers work on the language task ...

Many machine learning tasks, such as language modeling, may harbor hidden forms of periodicity and require periodicity modeling for OOD scenarios. For instance, consider addition and multiplication operations: despite being strictly non-periodic functions, they involve bit-level operations and carry mechanisms that exhibit periodic characteristics. Moreover, from language perspective, periodicity is not just a data feature but reflects a form of structural knowledge—one that allows for the transfer and reuse of abstract rules and principles across different contexts. Therefore, FAN’s ability to capture the latent periodic features contributes to its enhanced performance in these tasks to some extent. The conclusion stems from our experiments, where using FAN exhibits clear improvement especially for OOD scenarios in extensive experiments on Symbolic Formula Representation, Time Series Forecasting, Language Modeling, and Image Recognition tasks.

... the performance of KAN on general fitting tasks like fitting non-periodic symbolic functions or language modelling. But, if this is really the case, why would this be true? ...

We didn't compare our approach with KAN on Language Modeling tasks, since KAN's paper only conducted experiments on Symbolic Formula Representation tasks, and work [1] claims that "except for symbolic formula representation tasks, MLP generally outperforms KAN."

In Symbolic Formula Representation tasks, when the parameters are small, our approach achieves competitive results with KAN. However, when the parameters are large, our approach significantly outperforms KAN, as KAN shows a collapse and is even worse than MLP. In fact, the observation of KAN's performance collapse at large parameters is also described on Page 10 of KAN's paper, "... the test losses first go down then go up, displaying a U-shape, due to the bias-variance tradeoff ..." Figure 2.3 of KAN's paper also shows this phenomenon as the grid increases when fitting $f(x, y) = exp(sin(x)+y^2)$, which is included in our experiments as well. Moreover, as we know, some works also observe similar experimental results, such as Figure 8 of work [6] and Figure 6 of work [7].

Reference:
[1] KAN or MLP: A Fairer Comparison.
[6] Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving PDEs based on Kolmogorov Arnold Networks.
[7] On Training of Kolmogorov-Arnold Networks.

Thanks very much for your comments. We will reply to your questions point-by-point below and look forward to having further discussions with you!

This is my primary justification for my score. I don’t necessarily think that modifying the architecture of KANs and MLP with a different transcendental function is conceptually novel ...

We did not simply modify the activation function of KANs and MLP, and the architecture of FAN is different from them. Specifically,

First, we introduce Fourier principle into FAN's architecture. All signals or tasks in the world can be divided into the superposition or combination of periodic and non-periodic parts. Therefore, enabling general neural networks to model periodicity is necessary. However, existing general neural networks struggle to model the periodicity from data, especially for OOD scenarios, indicating their potential flaws in the modeling and reasoning of periodicity. In this paper, by introducing Fourier principle, the periodicity is naturally integrated into the structure and computational processes of FAN, thus achieving a more accurate expression and prediction of periodic patterns.

Second, we address the problems of traditional Fourier Neural Networks, i.e., approximate the Fourier coefficients are independent of the depth. We design FAN following the two key principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth, i.e., performing better as depth increases; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers. Achieving both principles simultaneously is neither straightforward nor trivial. The first one enhances the expressive power of FAN for periodicity modeling by leveraging its depth, while the second one ensures that the features of FAN’s intermediate layers are available to perform periodicity modeling. As a result, FAN can serve as building blocks for deep neural networks.

Third, FAN has fewer parameters and FLOPs. As shown in Table 1, FAN reduces the number of parameters and FLOPs by sharing the parameters and computation of Sin and Cos parts. Therefore, with the same input and output dimension, FAN has fewer parameters and FLOPs than MLP and KAN (KAN has more parameters and FLOPs than MLP with the same settings [1]), which also proves that FAN is not simply modifying the transcendental function.

Fourth, FAN has good performance on general tasks, especially for OOD scenarios. Existing Fourier Neural Networks are usually designed as the specialized model for a single task. However, we demonstrated the effectiveness of FAN with extensive experiments on real-world applications, including Symbolic Formula Representation, Time Series Forecasting, Language Modeling, and Image Recognition tasks (updated in the revised version, see anonymous github for detailed results).

Table1: Comparison of MLP and FAN, where $d_\text{p}$ defaults to $\frac{1}{4}d_\text{output}$, $d_\text{input}$ and $d_\text{output}$ denote the input and output dimensions of layer. In our evaluation, FLOPs for any arithmetic operations are considered as 1, and for Boolean operations as 0.

Reference:
[1] KAN or MLP: A Fairer Comparison.

For a higher score I would either need a motivating application, perhaps a SciML problem where the Fourier function class is needed ...

We conduct experiments on the SciML problem that includes the Fourier function class following the work [2]. The Burgers’ equation, a non-linear partial differential equation, is frequently used in scientific computing to model shock waves and traffic flow, among other phenomena. The detailed error rate on Burgers’ equation is listed in the Table below, where the values represent the Average Relative Error and the lower values indicate better performance. We can find that replacing the MLP Layer with FAN Layer in Fourier Neural Operator (FNO) can achieve clear improvements on each setting of resolution s of this task.

Reference:
[2] Fourier Neural Operator for Parametric Partial Differential Equations.

Dear Reviewer D8kJ,

Thanks for your suggestions. We are glad that you can see the value of FAN. However, we also would like to clarify that we conducted extensive experiments to verify the effectiveness of FAN, specifically: 1) We verified the powerful periodicity modeling capabilities of FAN across various periodicity modeling tasks, as shown in Figure 1, Figure 3, Figure 6, and Figure 9. 2) We verified the effectiveness and generalizability of FAN in extensive real-world tasks, including symbolic formula representation, time series forecasting, language modeling, and image recognition. 3) We have added the Further Analysis of FAN in Section 4.6, including the impact of hyperparameter $d_p$ and the runtime of FAN, providing deeper insights into its efficiency. 4) We have also updated additional experiments in Appendix B.1 to B.7, e.g., the SciML problem you mentioned and further experiments on time series problems.

Regarding time series forecasting tasks, the most frequently studied public datasets, such as the Weather, Exchange, Traffic, and ETTh datasets [8, 9, 10], have been employed in our experiments, and to the best of our knowledge, the related works didn't conduct experiments on physiological data or EEGs. For physiological data or EEGs, can you specify some public benchmark or related work of time series forecasting that conducts experiments on physiological data or EEGs?

Finally, to address the poor extrapolation of these general neural networks such as MLP and Transformer in periodicity modeling, our work designs a general model building block. For general real-world tasks (like language modeling and image recognition), using FAN achieved competitive or superior performance than the baselines, especially for dealing with OOD scenarios, since the real-world tasks inherently contain many periodic and non-periodic features, although some of them are hidden.

We look forward to your feedback and further discussions with you.

Sincerely,
Authors

Reference:
[8] Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. NeurIPS
[9] Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI
[10] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. ICML

All of the mathematical formulas in your comment are garbled, significantly impairing the readability and comprehension. For examples, "...the distinction between  and  is unclear...", "... include both low and high frequencies (e.g., )", "Explain the effect of the parameter  on model performance.", and "Provide details on how  influences the results.". Please correct these mathematical formulas to enhance the clarity. We will try our best to answer your questions below. Moreover, some of the questions you raised were already addressed in our paper, which you may miss, and we will restate them. We look forward to further discussions with you.

Comparing with frequency-based (FFT-based) models in the Time Series Forecasting tasks.

We did not compare with these specialized models because we aim to design a general model building block, which addresses the poor extrapolation of these general neural networks such as MLP and Transformer in periodicity modeling. This is also the reason we conduct the extensive experiments on Symbolic Formula Representation, Time Series Forecasting, Language Modeling tasks, and Image Recognition tasks (updated in the revised version, see anonymous github for detailed results) to prove the effectiveness of FAN. These specialized models are hard to cover all these tasks.

For the frequency-based models in Time Series Forecasting tasks, we recommend you employ them simultaneously, i.e., replace MLP with FAN in frequency-based models, to achieve better results. We present the experimental results below, where the results of FEDformer are cited from its paper directly. From the results, we can find that FEDformer with FAN can outperform FEDformer in almost all cases.

Clarify the distinction between your approach and directly learning the coefficients.

Without well-definition of "Directly learning the coefficients", we find it difficult to respond to this question. Our FAN Layer models the Fourier coefficients with the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers. The first one enhances the expressive power of FAN for periodicity modeling by leveraging its depth, while the second one ensures that the features of FAN’s intermediate layers are available to perform periodicity modeling.

Does "Directly learning the coefficients" in your opinion mean inputting sin(x) and cos(x) and then using MLP Layer instead of FAN Layer to model the Fourier coefficients? In this setting, frequencies are fixed and only the coefficients are learned, which may limit the model's ability to capture patterns not aligned with these frequencies. Taking simple f(x) = x mod 5 as an example, this setting may not even converge at all, because the frequency of x mod 5 is inconsistent with sin(x) and cos(x). The experimental results of their loss are shown as follows.

Conduct an ablation study to quantify the improvement in results when incorporating the linear component.

We disagree. Do you mean removing the linear component in sin and cos part, i.e., changing the expression from $[\cos(W_px)\|\| \sin(W_px)\|\| \sigma(B_{\bar{p}} + W_{\bar{p}}x)]$ to $[\cos(x)\|\| \sin(x)\|\| \sigma(B_{\bar{p}} + W_{\bar{p}}x)]$? We are uncertain why we need to compare with it, as it is obviously a specific case of our approach. If $W_p$ is the identity matrix, then the representational effects of both approaches become equivalent. However, our approach offers the flexibility to learn more customized combinations of the previous layer’s features. Moreover, there appears to be a potential issue with dimension alignment. This modification would make the output dimension at least twice the input dimension, leading to exponential growth in dimensions with increased network depth.

The reason we didn't conduct the ablation study was that the operations within our FAN Layer are integrated as a whole. Removing any one of these operations would violate the above-mentioned principles and cause the entire network not to work normally.

Thank you for your comments. We appreciate that you recognize the FAN's value. The experiments you mentioned have been conducted in our paper or explain the unreasonability in the response. Could you explain what aspect of experiments you require to further demonstrate the novelty or performance of FAN? (We would also be grateful if you could correct the garbled mathematical formulas of your original responses to help us understand.) We would like to clarify that we conducted extensive experiments to verify the generalizability and effectiveness of FAN, specifically:

• We verified the powerful periodicity modeling capabilities of FAN across various periodicity modeling tasks, as shown in Figure 1, Figure 3, Figure 6, and Figure 9.
• We verified the effectiveness and generalizability of FAN in extensive real-world tasks, including symbolic formula representation, time series forecasting, language modeling, and image recognition.
• We have added the Further Analysis of FAN in Section 4.6, including the impact of hyperparameter $d_p$ and the runtime of FAN, providing deeper insights into its efficiency.
• We have updated the additional experiments in Appendix B.1-B.7, including further experiments on time series problems, FAN for solving SciML problems, the effectiveness of FAN Layer for deep neural networks, and so on.

We look forward to your feedback and further discussions with you.

Thanks very much for your comments. We will reply to your questions point-by-point below and look forward to having further discussions with you!

The paper lacks theoretical justification for why FAN enhances generalization capability.

From a qualitative perspective, real-world tasks inherently contain many periodic and non-periodic features (although some of these features are hidden). However, existing general neural networks struggle to model the periodicity from data, especially for OOD scenarios, indicating their potential flaws in the modeling and reasoning of periodicity. By introducing Fourier principle, FAN addresses the aforementioned challenges and has the ability to capture and model both periodic and non-periodic features, so FAN can enhance generalization capability. We provided this analysis in our Discussion Section.

In this paper, we provide extensive experiments to demonstrate the generalizability of FAN, which achieves clear improvements on a range of real-world applications, including Symbolic Formula Representation, Time Series Forecasting, Language Modeling, and Image Recognition tasks (updated in the revised version, see anonymous github for detailed results).

They don't explain how FAN avoids overfitting, such as through some model selection methods.

We didn't adopt any specific methods to prevent overfitting for FAN. All settings are consistent for FAN and the baseline, i.e., we train them for a predetermined number of epochs. In Figure 4, we also displayed the training and testing loss of different models throughout the process of learning complex periodic functions. From Figure 4, we can observe that the performance of FAN significantly outperforms other baselines across all epochs, both in training and testing loss.

FLOPs analysis is given in this paper, but the comparison of actual training time is lacking

Due to PyTorch’s optimization of MLP, although FAN has lower FLOPs, the actual training time of replacing MLP with FAN is similar. We conduct another experiment to show the advantages of FAN's lower FLOPs below, which will be added to our revised manuscript. We can find that MLPs exhibit smaller runtimes when the input and output sizes are small, due to PyTorch’s optimization of MLP. However, as the input and output sizes continue to increase, matrix computations become the main contributor to runtime. At this point, FAN's fewer parameters and reduced FLOPs begin to show significant advantages. Note that FAN can be further optimized from the underlying implementation, we leave this to future research.

Will the result of a deeper FAN yield a better fit? Will overfitting results be produced? It is recommended to test the performance changes of layers 3-20 at different depths.

Yes, a deeper FAN usually yields a better fit in our experiments. We have evaluated the effect of varying the number of FAN layers from 3 to 20 on periodicity modeling tasks, employing residual connections to mitigate overfitting, as shown in Figure 8 of Appendix. The experimental results show that both the best training loss and test loss still decrease slowly as the number of layers increases.

Furthermore, on Language Modeling tasks, we replaced 24 MLP Layers of Transformer with 24 FAN Layers, i.e. Transformer with FAN, and it also achieved clear improvements on each task, especially for OOD zero-shot evaluation scenarios. These findings indicate that FAN Layer is effective for deep neural networks.

This article compares performance with traditional MLP, KAN and transformer. Whether to compare and discuss with deep learning frameworks that actually deal with time series data, such as LSTM, RNN

We compared with LSTM (a well-known RNN) on Time Series Forecasting and Language Modeling tasks, as demonstrated in Tables 2 and 3, respectively.

Most of your questions have been addressed in our original paper, and we will restate these arguments. We reply to all your questions point-by-point below, hoping they can answer your questions, and look forward to your high-quality replies and following discussions with you.

1.Lack of detailed explanation about parameter efficiency: The abstract claims that FAN can substitute MLP with fewer parameters. However, the theoretical justification and experimental evidence supporting this claim are not sufficiently detailed in the main text. ...

We disagree. We already demonstrated the parameter efficiency of FAN in our original paper through both theoretical analysis and experimental results. In theoretical analysis, FAN reduces the number of parameters and FLOPs by sharing the parameters and computation of Sin and Cos parts. As shown in Table 1 of our paper, we provided the theoretical differences in both parameters and FLOPs, and their main difference is the preceding coefficient (i.e. $(1-\frac{d_p}{d_\text{output}}) < 1$ ) in FAN Layer, so FAN can substitute MLP with fewer parameters and FLOPs. Moreover, in our experiments, we aslo presented the comparisons of parameter counts in various tasks, such as Time Series Forecasting tasks (Table 2), Language Modeling tasks (Table 3), and Symbolic Formula Representation tasks (Figure 5).

Table1: Comparison of MLP layer and FAN layer, where $d_\text{p}$ is a hyperparameter of FAN layer and defaults to $\frac{1}{4}d_\text{output}$ in this paper, $d_\text{input}$ and $d_\text{output}$ denote the input and output dimensions of the neural network layer, respectively. In our evaluation, the FLOPs for any arithmetic operations are considered as 1, and for Boolean operations as 0.

2.Doubtful results in symbolic formula representation: In Figure 5, the performance of KAN was not replicated accurately. As KAN has shown strong results in symbolic formula representation in previous works, the omission of this accuracy raises concerns. ...

We wholeheartedly disagree. The experimental results in symbolic formula representation are correct and reproducible, as we directly inherited the code from KAN's project, and our results are consistent with the observations made in KAN as well as some related studies，specifically:

As stated in our first paragraph of Symbolic Formula Representation Section, "We follow the experiments conducted in KAN’s paper, adhering to the same tasks, data, hyperparameters, and baselines. In addition to the original baselines, we also include Transformer for comparison in this task." We directly inherited the code from the KAN project for our experiments, and all results are reproducible using the code we provided. We have further thoroughly verified the code and confirmed there is no problem within it.

In Figure 5, we compare our approach with the baselines across varying numbers of parameters on symbolic formula representation tasks. Following KAN's paper, the parameter size of KAN is increased by adding grid points $G$ with a fixed architecture they provided (Details can be found on Page 14 of KAN's paper), and for other baselines, the parameter size of the i-th layer baseline is increased by adding hidden size.

Comparing the experimental results—Figure 5 in our paper and Figure 3.1 in KAN’s paper—you can find that all baseline results are consistent. Figure 3.1 only provided KAN's partial results in the range of small parameters, and complete results for the remaining baselines. In Figure 5 of our paper, we presented the complete results of all approaches.

In fact, the observation of KAN's performance collapse at large parameters is also described on Page 10 of KAN's paper, "... the test losses first go down then go up, displaying a U-shape, due to the bias-variance tradeoff ..." Figure 2.3 of KAN's paper also shows this phenomenon as the grid increases when fitting $f(x, y) = exp(sin(x)+y^2)$, which is included in our experiments as well. Moreover, as we know, some works also observe similar experimental results, such as Figure 8 of work [1] and Figure 6 of work [2].

Reference:
[1] Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving PDEs based on Kolmogorov Arnold Networks.
[2] On Training of Kolmogorov-Arnold Networks.

Dear Reviewer 7Gdo,

Thank you for your feedback. We are pleased to hear that your technical concerns have been addressed. However, regarding your comment on the fundamental contribution of the paper, we find it difficult to understand what specific areas of our work require to further strengthen, as the feedback seems somewhat general. We would greatly appreciate it if you could kindly provide more detailed guidance on the aspects that you feel need improvement, so that we can make the necessary revisions and clarify any remaining concerns.

We look forward to your feedback and will make every effort to ensure the paper meets the expectations for publication.

Sincerely,
Authors

We are pleased that the reviewers' technical concerns have been addressed and their recognition of FAN's value during the rebuttal period. We appreciate that if the reviewers could offer any specific suggestions for further enhancing our work or consider updating the score if all questions have been resolved. Thanks for the time and effort of the reviewers and chairs!