Fourier Ordinary Differential Equations

We appreciate the reviewer's recognition of our work. We also thank you for the feedback regarding the experimental validation of our study.

We have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks. For a fair comparison, FNO is conducted with one Fourier layer where modes=2 and width=8. Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

[1] Kidger, P., Morrill, J., Foster, J. and Lyons, T., 2020. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, pp.6696-6707.

[2] Li, Zongyi, et al. "Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2020

We appreciate the questions the reviewers raised. We'd like to provide further clarification and insight into these questions.

Q1: Euclidean space in our paper is just used to contrast to the frequency space. We realized it may cause confusion, so we changed it to “Time domain” in the revision.

Q2: We appreciate the review’s question but we do not refer to convolution property here. We aim to say When one transforms a signal into the Fourier domain (using the Fourier Transform), differentiation with respect to time becomes a multiplication by the frequency variable. Mathematically, if f(t) is a time-domain function and F(ω) is its Fourier transform, then the Fourier transform of the derivative df(t)/dt​ is given by jωF(ω), where j is the imaginary unit and ω represents frequency. It is actually an important motivation for our work. Based on these properties, we plan to learn the derivative df(t)/dt​ in the Fourier domain and then use it for ODE based model.

Q3 & Q4: Thanks for pointing them out. Yes, these are some typos, and we corrected them in the revision.

Q5: In our setting, the DFT(FFT) from equ (4) is being performed along the time series length dimension. Thus, I think the frequency of it will be easily understandable.

Q6: We realize that compared with frequency space-based method is important to support the superiority of the FODE model. we have conducted additional comparative experiments to support the superiority of the FODE model. We further compare FODE with FNO[1] (belongs to the frequency space-based method) and Neural Controlled ODE (Other reviewers mentioned) on time series classification tasks. We read [2] and we think it is a good baseline, we added [2] in related work since the time limit. For a fair comparison, FNO[1] is conducted with one Fourier layer where modes=2 and width=8. Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

Q7: This is a typo. For time-series classification, we use Cross-Entropy Loss as the loss function, and the test loss is what we recorded during the experiment. Thus, table 3 describes the “Cross-Entropy Loss ”. Please see the revision.

Q8: The NODE with comparable parameters will still have worse performance than FODEs. We can be convinced of this point from our rich empirical experiments. Also, even ANODEs and SONODEs are more advanced models and always have better performance than original NODEs with comparable parameters. We also compared more models as shown in Table 3.

Q9: From Figure 4, we can see how hidden states evolve in the frequency domain for time series. The short-time Fourier transform is utilized here to enhance the visualization of the signal’s frequency domain Information.

Q10: Figure 5 is used to illustrate how the filter K be trained during training. The evolution of filter K with different initialization. From left to right, K is initialized by Zeros, Ones, and Xavier Uniform, respectively. From top to bottom, each subfigure shows the timeline of the training process (the training loss is from high to low). The changing of colors represents the changing of weight values in K during training. We added more descriptions in the revision.

[1] Li, Zongyi, et al. "Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2020

[2] Holt, Samuel I., Zhaozhi Qian, and Mihaela van der Schaar. "Neural Laplace: Learning diverse classes of differential equations in the Laplace domain." International Conference on Machine Learning. PMLR, 2022

We appreciate the questions the reviewers raised. We'd like to provide further clarification and insight into these questions.

Q1: For the time series forecasting tasks, MSE is used as the loss function. For the time series classification tasks, the Cross-Entropy Loss is used as the loss function. We recognize the importance of providing comprehensive details about our experimental procedures. We provide more experimental setup details in the revision, please see the environment setup section.

Q2: we have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks. For a fair comparison, FNO is conducted with one Fourier layer where $modes=2$ and $width=8$. Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

Q3: We agree with the reviewer that how well the model scales to more complex data is an important question. In this work, we plan to demonstrate a new method that learns the derivative df(t)/dt​ in the frequency domain and we test two tasks on several datasets to show this idea can work. We will test more complex datasets in our future work.

Q4: There are several advantages of the FODE over NODE. First is the accuracy, as shown in Table 1 and Table 3, the performance of FODE is much better than NODE. Second, the FODE provides an innovative way to calculate the derivative df(t)/dt​ in the frequency domain which may be an efficient way to capture the long-term dependence of time series data, and our experiments verify it.

Q5: We did not record the training and inference time of FODE and other models. From our experience, basically, the training time of FODE will be a little more than NODE, depending on the specific datasets.

Q6: In our paper, there are two experiment tasks. One of them is time series forecasting, as the reviewer mentioned. In the time series forecasting tasks, we try to predict future values based on these historical values. The number of future values and historical values can be more than 1, for example, we use 50 historical values to forecast values in the future. The reason we use a sliding window approach is to provide a more flexible way to divide the data for this task. When the only data point is used, the FFT transform can be applied to the feature dimension. However, if the feature dimension is 1, our model currently cannot handle this situation. But we left the forecasting task based on just one initial point as a future work.

[1] Kidger, P., Morrill, J., Foster, J. and Lyons, T., 2020. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, pp.6696-6707.

[2] Li, Zongyi, et al. "Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2020

1. Thank you for raising these important concerns. We carefully considered these questions during the design of our model. We carefully considered these questions during the design of our model. We acknowledge that non-periodic and noisy signals can pose challenges to the FODE model's performance and applicability. To address these issues, we incorporated an element-wise filter into our model's architecture. The element-wise filter is a crucial component of our approach, designed to enhance the model's ability to handle various types of time series data, including those that are non-periodic or have high levels of signal noise. This filter allows the FODE model to adapt and capture the underlying dynamics even in challenging conditions.

2. We appreciate your concerns regarding the computational efficiency of our approach, and we'd like to provide further clarification and insight into our efforts to address these issues. (1) we use the FFT and IFFT inside the “ODE Function” and it makes sure the dimensions of input and output dimensions remain the same with a Neural ODE. For example, if the input data has a shape of (d1, d2) = (length, #features) after applying the FFT, the tensor’s shape becomes (d1’, d2’) = (length, #features//2), where d2’ contains real part and Imaginary part. (2) we use the Fast Fourier Transform (FFT) to reduce the computational complexity of DFT. (3) the integration of FFT and IFFT inside the 'ODE Function', coupled with their interaction with the ODE solver, forms a synergistic relationship. This synergy not only improves computational efficiency but also aids in effectively learning the vector field. As a result, it can potentially reduce the number of function evaluations required, further contributing to the model's overall efficiency.

3. Thank you for your feedback regarding the experimental validation of our study. (1) we recognize the importance of providing comprehensive details about our experimental procedures. we provide more experimental setup details in the revision, please see the environment setup section, including the number of epochs, loss function, ODE solver method, details of layers, etc. Algorithm 1 also shows the detailed implementation of our model. (2) we have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks. For a fair comparison, FNO is conducted with one Fourier layer where modes=2 and width=8. Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

[1] Kidger, P., Morrill, J., Foster, J. and Lyons, T., 2020. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, pp.6696-6707. [2] Li, Zongyi, et al. "Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2020