Fourier Analysis Network

Thank you for your insightful comments. We will respond to your comment point-by-point below.

I believe FAN represents a valuable contribution for modeling periodic functions, but I struggle to see how it qualifies as a general-purpose architecture given the experimental evidence.

Thanks for recognizing the value of our work. Compared with existing Fourier-based networks, which are typically designed for specific tasks, we have explored the capability of FAN to be applied to more types of tasks. Based on the current experimental results, we observed that FAN has the potential to be applied to a range of real-world tasks. However, whether FAN equips the general-purpose capabilities remains a question to be answered. We will adjust the statements like "maintains general-purpose modeling capability" to more precise descriptions, such as "maintains capability to be applied to more types of tasks" in our revised manuscript.

Could the authors clarify their vision for FAN's practical applications? Specifically, which real-world tasks would benefit from the Fourier-based inductive bias, and how would you recommend identifying when FAN layers would be preferable to standard MLPs? & I'd recommend the authors include a more thorough discussion of the limitations in their real-world task selection, specifically addressing why certain tasks were chosen and which types of applications might not benefit from FAN.

That's a good question. Periodicity is a fundamental and ubiquitous characteristic in both the natural sciences and human activities. However, existing general-purpose neural networks, such as MLPs and Transformers, exhibit inherent limitations in effectively capturing and modeling these periodic features, while existing Fourier-based networks face challenges in scaling to deeper networks and are typically designed for specific tasks. To this end, we propose FAN to effectively address periodicity modeling challenges while maintaining capability to be applied to more types of tasks.

Various forms of real-world data contain either explicit or implicit periodic patterns. We believe that these real-world tasks will benefit from FAN. When the task is completely aperiodic, FAN degenerates to MLP. Although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), the boundaries of this model's generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work. We have added this discussion to our revised manuscript.

The paper compares against vanilla architectures rather than specialized state-of-the-art models. If FAN layers can serve as replacements for MLPs in existing architectures, would current specialized SOTA actually benefit from this substitution?

Yes, we hope it can replace MLP in some aspects, and we attempted to demonstrate this in our paper (due to space and experimental limitations, we only explored its advantages on existing tasks). As shown in Appendix B.3 and B.4, we compared SOTA methods with using FAN in Large Language Modeling scenario and Solving SciML Problems scenario, where it outperforms specialized SOTA methods.

Sentiment analysis seems like an odd choice to showcase FAN's language modeling benefits.

Thanks for your comment. Our intention with the experiments was twofold. First, our primary goal was to validate FAN's effectiveness in the fundamental task of language modeling, which we demonstrated through its ability to achieve a lower entropy loss. This part of the experiment directly addresses its language modeling capabilities. Second, we wanted to conduct an exploratory investigation into whether FAN has the potential to be applied to higher-level, downstream NLP tasks, such as sentiment analysis (one of the most widespread real-world NLP applications, used for analyzing user reviews, social media sentiment, and brand monitoring). If you think this result is odd, we are happy to remove the sentiment analysis results, with only maintaining the results of language modeling, in the camera-ready version.

To improve clarity, I’d suggest moving the description of the FAN(Gated) variant to the main paper.

Thank you for your suggestion! We have moved the description of the FAN(Gated) variant to the main paper in our revised version.

Thank you for addressing my concerns! I have a better understanding now of why you see FAN as a general replacement for MLP. About the sentiment analysis experiment, it is an interesting result to include in the main paper, but it's also interesting to see how the method performs on other LLM tasks. I'd suggest directly referring readers to Appendix B.3 for more results. Overall, I agree that vast analysis of FAN performance on general tasks can be conducted in future work. I increased my score.

Dear Reviewer rvp8,

Thank you for your encouraging feedback and constructive suggestions! We sincerely appreciate the time and effort you’ve dedicated to reviewing our work. Your insightful comments have greatly strengthened the paper, and we’re grateful for your support in raising the score.

Best regards,
Authors

Thank you for your thorough review of the related literature and detailed feedback. We will respond to your comment point-by-point below (we merged some similar comments).

Q1: The work happily cites work ... yet fails to properly compare -- theoretically, experimentally, or both -- to contemporary works many of which have proposed their own methods for the same purpose.

Thanks for your comment. We had cited works [1-5] and compared with them in our original submission, and we are grateful for your pointing out reference [6], which we have added the discussion of [6] in our revised manuscript. Our work is largely different from this work [1-6], specifically:

For work [1-4], they face challenges in scaling to deeper networks, i.e., the capacity of their deep networks to fit the Fourier coefficients is independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling. Our core differences are, "we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers." We introduced and categorized them as Fourier Neural Networks (FNNs) [1-4] in Related Work Section. In Section 4.3, we conduct experiments to compare our approach with FNNs (we will detailed discuss the difference between FAN and FNNs in Q2), and FNNs cannot fit language modeling tasks but our approach works well.

For work [5,6], they focus on different purposes from our work (we will detailed discuss in Q3). And work [5] assumes all input signals have the period of 1 (as stated in page 3 of work [5]), which we conducted experiments on the same setting in Appendix B.6, and it cannot fit our periodicity modeling tasks. Work [6] is a comparative study without proposing new method, and RNN performs well in the setting of work [6], but also fails in our periodicity modeling tasks (detailed results are shown in Q6).

Sorry for the misunderstanding. To enhance the clarity, we have enhanced the connection between them and added this discussion to our revised manuscript.

Q2&Q4: Closely related variants of the FAN proposed in Section 3... On lines 125-130, the point of comparison should not have been MLP...

In our paper, we compare our approach with not only MLP but also previous Fourier-based networks in experiments of Section 4.3. The reason why we formally analyze the differences with MLP in Section 3 and lines 125-130 is that our goal is to explore the likelihood of FAN to offer broad applicability like MLP.

We provide the analysis of work [1-4] compared to our approach in below table and have added it to the appendix of our revised version. We mainly discuss work [1-3], due to the work [4] is a comparative study without proposing new method. The work [2, 3] can be regraded as the simplified version of work [1], where work [2,3] are fully equivalent to work [1] when hyperparameter m=1 (They can be converted to each other through trigonometric identities, and the differences in their parameters can be absorbed by the learnable W and B). The work [5] cannot fit our periodicity modeling tasks, so we didn't compared with it.

Notation Key:

• $d_i = d_{\text{input}}$ (Input dimension hyperparameter)
• $d_o = d_{\text{output}}$ (Output dimension hyperparameter)
• $d_p$ = FAN layer hyperparameter (default $\frac{1}{4}d_o$)
• $d_h$ = Hidden dimension hyperparameter (for [1], [3])
• $d_c, d_s$ = Layer [2] hyperparameters (cosine/sine branch dimensions)
• $m$ = Layer [1] hyperparameter (number of projections)
• $\gamma$ = FLOPs per nonlinear activation ($\sigma$, $\cos$, or $\sin$)

Q3 & Q5: Figure 3 closely follows the exposition Figures 1&2 of [6]. Experimentation of Section 4.1, Appendix C, and Figure 4 mirrors...

Our work and work [6] focus on different problems, with only the display form looking like similar. Specifically, the model's input and output in Figure 3 are both scalars, i.e., function input and function output, and we show the results of entire test dataset. The input of the network in Figures 1 & 2 of [6] is the sequence of previous step, and the output is the value of the next step, and they show the median model prediction and the shaded regions after 30 runs of test sample. This distinction explains their use of RNN. While an RNN excels in the settings of [6], it performs poorly on our periodicity modeling task (See results in Q6).

Moreover, experiments in Section 4.1, Appendix D (Appendix C in Q5 is maybe a typo. Appendix C includes the experimental setup of real-world tasks, which is not related to work [5,6]), and Figure 4 are also different. First, Figure 4 analyzes the entire model training process from another perspective, proving that its inability to fit periodicity is not due to underfitting. There is no similar experiment in work [5,6]. Second, Section 4.1 and Appendix D includes the experiments of periodicity modeling task, which are different from the problems studied in work [5,6].

Q6: Experimentation of Section 4.1 glosses over RNNs...

We compared with LSTM, considering it is generally considered to be a powerful and effective variant in the RNN family. To address your concerns, we have supplemented the results of the standard RNN and another variant of RNN, GRU, below. The results show that our approach outperforms both RNN and GRU on various tasks, including periodicity modeling (the experiment in Section 4.1), time series torecasting, and language modeling.

Q7: Table 4 acknowledges the existence of [1] and FTNNs but ignores [2]-[5]...

We would like to clarify that we did not ignore work [2-5] in Table 4, where we compared the performance of our approach FAN, FNN, FSNN, FTNN, and MLP on language modeling tasks. Please note that FNN and FSNN are more capable generalizations of [1-4] and [5], respectively. As mentioned in Q2, work [2-4] can be regarded as a simplified version of FNN [1]. And work [5] can be regarded as a simplified version of FSNN, where it assumes all input signals have the period of 1. Therefore, in our comparison with FNN [1] and FSNN, we are, in fact, considering all the works [1-5].

Q8: The inadequacy of snake activations described on lines 255-258 ...

The comparison with snake in the appendix is not proposed as a new finding, but only illustrates its shortcomings. We have modified the statement and added the discussion of work [6] in our revised manuscript.

Q9: The authors do not answer truthfully in the Reviewer Checklist under point 2..

Thanks for your feedback. Due to space limitations in the main text, we only discussed the limitations of our work in the Justification of Checklist. We have added a section on limitations to our revised manuscript:

First, we only demonstrate the effectiveness of FAN on some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), and we aim to further broaden the applicability of FAN in our future work. Second, although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some real-world tasks, the boundaries of this model's generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work.

Dear Reviewer nmeh,

Do our responses address your concerns? Please let us know if there are any other questions you'd like to discuss! We are happy to address them while there is still time. Thanks again for your time and detailed review!

Best regards,
Authors

Thank you for your in-depth review and valuable feedback. We will respond to your comment point-by-point below.

Currently in a FAN Layer (Figure 2), so cosine and sine do not have the bias components (B_p), unlike the MLP part. Would adding the bias component help, for example, in modeling “phase shift” in periodicity? Note that after adding bias components, the network can use sine or cosine alone, without using both. In particular, how would such networks flair in experiments?

Thanks for your suggestions. We conduct experiments on the setting of adding bias components and use cosine alone, namely FAN_cos, and have added the experimental results to our revised manuscript. Overall, FAN outperforms FAN_cos on periodicity modeling and language modeling tasks. Although FAN_cos occasionally achieves lower training loss, its significantly higher test loss indicates poor generalization.

periodicity ratio (d_p) is yet another hyper-parameter to tune.

We fix the periodicity ratio to 1/4 and achieve stable and clear improvements across all experiments with this same setting. Although we found in the appendix that further adjusting the periodicity ratio can lead to even greater improvements for specific task, we believe that the default parameters used in this paper can already cover common scenarios.

Note on periodic generalization ...

We do believe that some generalization properties can be expressed as periodicity, so we aim to extract the information with periodic generalization using the Fourier Principle, which leads to FAN naturally performing better on tasks with periodicity. Of course, for real-world data, periodic extension is a valid and parsimonious assumption that complies with Occam's razor. However, the extent of its generalization capability is likely highly dependent on the specific real-world task. We have added this discussion to our revised manuscript.

Thanks for your comments. In the last rebuttal response, the Periodicity Modeling experiments was conducted on a complex compound periodic function $y = exp(\sin(\pi x)^2 + \cos(x) + x \mod 3 - 1$. To address your question about performance on shifted periodic data, we ran two new experiments: 1) $y = \sin(z + 0.5)$; 2) $y = \sin(w + 0.5) + \sin(2w)$. The results show that even on shifted periodic data, the generalization performance of FAN on the test data is much better than FAN_cos.

Moreover, we would like to provide a straightforward explanation for the finding in language modeling from the following perspective: Imagine that cos(Wx) and sin(Wx) are two fundamental "periodic building blocks", like red and blue LEGO bricks.

1. Periodic Part of FAN Layer ($[\cos(W_p x) || \sin(W_p x)]$) gives you both the red brick (cos) and the blue brick (sin) separately. The next layer of the network can then freely decide how much red and how much blue to use, allowing it to construct any combination of phase (the shade of color) and amplitude (the size of the brick). This is extremely flexible.
2. Periodic Part of Cosine-Only+Phase Shift ($[\cos(W_p x+B_p)]$) is like being given a pre-mixed "purple" brick, where the red and blue components have already been combined in a fixed ratio. The next layer can only decide whether to use this purple brick and with how much strength (adjusting its amplitude), but it cannot change the inherent "color mixture" (the phase) of the brick itself.

The core advantage of FAN is that the FAN layer gives the network greater freedom. By providing independent cos and sin building blocks, the network can more easily and precisely learn any periodic function it requires, much like a painter who has all the primary colors can mix any hue they desire. The "cosine-only + phase shift" method restricts this creative potential, making the learning process more difficult. This disadvantage becomes even more pronounced when stacking multiple layers to construct complex functions.

We hope our responses can address your problems. Thanks again for your dedicated time and insightful feedback.

Thank you for your thoughtful comment and constructive feedback. We will respond to your comment point-by-point below.

Additional analyses of layer activations and ablations could help to better understand why the approach works better than existing Fourier Series NNs. Would it be possible to do a layer-wise spectral analysis to illustrate how coefficients/frequencies evolve with depth?

Thanks for your suggestions. We conduct experiments on layer-wise spectral analysis and have added the experimental results to our revised manuscript. We perform a Fast Fourier Transform (FFT) on each layer's outputs and calculate four key metrics to quantify the spectral characteristics:

1. Spectral Centroid: Measures the "center of mass" of the spectrum, indicating whether the layer's features are concentrated in low or high-frequency regions.
2.  Spectral Sparsity (L1/L2 Norm): Quantifies how concentrated the spectral energy is within a few frequency bins. A higher value implies a more structured and less noisy signal.
3.  Spectral Entropy: Measures the uniformity and predictability of the spectrum. A lower entropy indicates a more ordered and well-defined spectral structure.
4.  Dominant Energy Ratio (Top-5): The proportion of total spectral energy contained within the top 5 most dominant frequency components, indicating how focused the representation is on key periodic features.

The results reveal a highly effective multi-stage learning process, which is more sophisticated than a simple monotonic evolution of frequencies. We observe a clear three-stage "Deconstruction-Exploration-Reconstruction" mechanism:

1. Initial Approximation (Layer 1): The first layer rapidly forms an initial, highly-focused approximation of the signal, as shown by its very high Dominant Energy Ratio (96.1%).

2. Feature Deconstruction and Exploration (Layers 2-8): To model the function's complex, non-sinusoidal components (especially the x mod 3 term, which requires a wide range of Fourier series terms), the intermediate layers must first "deconstruct" the signal. This is evidenced by a sharp increase in Spectral Entropy and a decrease in the Dominant Energy Ratio. The network actively disperses energy across a broader spectrum to explore and capture these challenging features, showcasing the flexibility afforded by its depth.

3. Integration and Reconstruction (Layers 9-11): In the final layers, the model's task shifts from exploration to integration. It "reconstructs" a final, efficient representation from the features learned in the middle layers. This is marked by a dramatic decrease in both Spectral Entropy and Spectral Centroid, alongside a sharp increase in the Dominant Energy Ratio to a final value of 93.8%. The network converges to a "clean", low-frequency, and highly structured representation that is optimal for the final linear layer to map to the target output.

Do FAN’s cosine/sine blocks hamper tasks that are intrinsically non-periodic? A negative result would be informative, and in my opinion, it would not reduce the value of the contribution.

FAN's cosine/sine blocks do not hamper non-periodic tasks. Specifically, non-periodic tasks can be categorized into three types: 1) Completely non-periodic tasks: Tasks with no periodic patterns at all. 2) Implicitly periodic tasks: Tasks that are mathematically non-periodic but can be viewed as periodic from a different perspective. For example, the operation y = a + b can be seen as the result of periodic operations of single-digit addition and carrying. 3) Partially periodic tasks: Tasks where only some components are periodic, such as a rising spiral. For implicitly and partially periodic tasks, which are common in the real world, FAN can achieve good results. For completely non-periodic tasks, FAN degenerates to a standard MLP, and we have not yet identified specific scenarios where it performs negatively. We leave this for our future work.

Section on prior work says: "Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks tends to be poorer" and lists a number of citations. It would be informative to expand that sentence and section further (perhaps in the appendix) and detail more precisely how the proposed approach differs from the prior work.

Thanks for your suggestions! We have expanded on this sentence as follows: Zuo and Cai (2005) proposed a control scheme based on Fourier Neural Networks, specifically for solving the tracking control problem for a class of unknown nonlinear systems. Tan (2006) studied the application of Fourier Neural Networks in the problem of aircraft engine fault classification. Jiang et al. (2022) utilized Fourier series to improve the accuracy of time series forecasting tasks. Chen et al. (2022) proposed a Fourier Imager Network that can perform end-to-end phase recovery and image reconstruction for raw holograms of novel samples. Uteuliyeva et al. investigated the poor performance of Fourier Neural Networks on general tasks such as language modeling. These works are often task-specific, and it has been proven difficult to handle more general tasks. We have added the detailed introduction of each citations to the appendix of our revised manuscript.

How the proposed approach differs from the prior work: Previous Fourier-based networks face challenges in scaling to deeper networks, i.e., the capacity of their deep networks to fit the Fourier coefficients is independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling. Our core differences are, "we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers." In Section 4.3, we categorized these Fourier-based networks into three main types and compared with them, the experimental results show that only our approach achieves superior performance on both periodicity modeling task and language modeling task.

I suggest doing some ablation experiments to better understand what are the key contributors in FAN. For instance, freezing W_p or having just sine or cosine function or having FAN layers only in part of the network.

Thanks for your suggestions. We conduct experiments on just cosine function and having FAN layers only in part of the network, and have added the experimental results to our revised manuscript. The results show that our current approach demonstrates a clear advantage over the variants in Periodicity Modeling and Language Modeling tasks.