Neural Fractional Attention Differential Equations

I. New Supporting Experimental Results

We appreciate the reviewer’s critique of the experimental evaluation and thank them for motivating us to test our framework on more complex, non-linear, temporally evolving dynamical systems. The marginal advantage reported in the original paper may be attributable to the evaluated system not being sufficiently challenging to fully demonstrate the strengths of our framework.

To further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.

1. Fluid Flow Prediction:

We evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution. Turbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in Table R1, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches.

Table R1: Average prediction error of different models on turbulent vector field prediction.

[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.

2. Urban Population Mobility Prediction:

Post-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2]. This dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in Table R2, KatFDE achieves state-of-the-art performance. This is a quintessential use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. The model can down-weight pre-disaster commuting patterns during the hurricane and re-integrate them during recovery.

Table R2: Average prediction error of different models on post-disaster urban mobility.

[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.

3. Biological Neural Spike Train Dynamics:

We next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context. We tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification. As Table R3 shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems.

Table R3: Test Accuracy of different models on neuron spike train classification (%)

[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.

---

II. Ablation Study: If Remove Learnable Attention:

We emphasize that the comparisons in Table 1 already provide a clear ablation analysis. This is fully consistent with our explanation of the novelty, which is framed relative to the first-order derivative, the constant-order Caputo derivative, and the variable-order derivative definitions. Each of these progressively introduces greater advantages and flexibility to the continuous system. Our framework fully generalizes and unifies all of these approaches.

The ablation study is shown in Table R4, reproduced from Table 1 in the paper.

Table R4: Ablation study among differential-equation-based continuous models. The results show that through generalization, our model achieves an advantage.

---

III. Interpretability of the Kernel

The complex fully learnable kernel is inherently difficult to interpret. We will include attention heatmaps in the revised version to provide better visualization. Several traditional fractional derivative operators (see Eq10. and Eq12. in the paper) define

$$K(t, \tau, \mathbf{x}(t), \mathbf{x}(\tau))=\frac{\psi^{\prime}(\tau)(\psi(t)-\psi(\tau))^{\alpha-1}}{\Gamma(\alpha)} \quad \text { or } \quad K(t, \tau, \mathbf{x}(t), \mathbf{x}(\tau)) = \frac{(t-\tau)^{\alpha-1}}{\Gamma(\alpha)},$$

which depend only on the temporal distance between states. In these cases, closer states exert a stronger influence on each other than more distant ones. For a general kernel $K$, however, direct interpretability is considerably more challenging.

---

IV. Comparison with Informer/FEDformer/ContiFormer

As demonstrated before, this work aims to propose an innovative framework that is flexible and scalable for a wide range of applications. Within this framework, we verify the performance of the fusion of FDE model and scaled dot product attention mechanism. Informer[w1], FEDformer[w2] and ContiFormer[w3] are proposed to improve transformer either by reducing the complexity from $O(L^2)$ to $O(L\log L)$ or even $O(L)$ with ProbSparse self-attention or frequency enhanced attention with Fourier/wavelet transform respectively, or using averaged continuous inner product instead of dot product in transformer to tackle irregular time series. These advanced strategies are essentially compatible with our framework. Specifically, one can readily use attention mechanism proposed in Informer, FEDformer and ContiFormer by specifying corresponding kernels in (20). Therefore, we will work with Informer,FEDformer and ContiFormer in the current framework, instead of comparing with them.

[w1]Zhou Haoyi, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI2021.

[w2]Zhou tian, et al. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, ICML2022.

[w3] Chen Yuqi,et al. ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling, NeurlPS2023.

Experimental details

Some experimental details are presented in the appendix, and more will be added for completeness. The codebase for reproducing our experiments with exact parameters will be fully public. For brevity, we describe the setup for the node classification task on graphs. The experiments are implemented using the PyTorch framework on a single NVIDIA RTX4090 24GB GPU. For different datasets, parameters such as time steps and evolution time are slightly adjusted. Taking the Cora dataset as an example, we set t = 25, step size = 1, learning rate = 0.01, weight decay = 0.05, epoch = 800, and dim = 256. Following [48, 18], the model architectures of KatFDE-l and KatFDE-nl are provided in Appendix F. The visualization results are shown in Appendix I. If the paper is accepted, we will move the visualization results into the main text. Furthermore, we will include attention heatmaps in the revised version to provide better visualization, as suggested by Reviewer jCMz.

---

Computational complexity and memory usage.

The main focus of this work is to propose a unified framework of FDE methods from a new perspective and to verify its effectiveness. This framework opens new possibilities, such as fusing the attention mechanism with the memory kernel of FDE, which is investigated in this work. We focus only on the standard attention mechanism. Therefore, the extra computational complexity and memory usage from the attention mechanism are inevitable and standard.

To further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems (we refer the reviewer to the full results presented in the top rebuttal to Reviewer ikK2). Here we present the fluid flow prediction task with accuracy and memory consumption comparison:

We evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution. Turbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics.

The preliminary prediction results are summarized in Table R5, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches. The GPU memory usage of KatFDE is slightly higher than the baseline LSTM and ODE models, but comparable to the Transformer. This is expected since the extra computational complexity and memory usage from the attention mechanism are inevitable and standard. Importantly, with nearly the same GPU memory usage, our continuous KatFDE framework performs much better than the standard Transformer.

To alleviate GPU memory usage, due to the flexibility of KatFDE, one can incorporate efficient attention mechanisms such as [4], which can reduce the computational cost from $O\left(L^2\right)$ to $O(L \log L)$ in practice. However, investigating this type of improvement and other variants is beyond the scope of this work and will be left for future research.

Table R5: Average prediction error and memory consumption of different models on turbulent vector field prediction.

---

Advantages of KatFDE compared to the references

Recall that KatFDE is built from a general integral equation that explicitly incorporates the attention mechanism [3], while DDGCRN [1] and Hierarchical‑Attention‑LSTM [2] are not equation-based and rely on LSTM architectures. The advantages of KatFDE for tasks like traffic forecasting and the new tested spatio-temporally continuous non-linear temporally evolving systems (we refer the reviewer to the full results presented in the top rebuttal to Reviewer ikK2) are threefold: i) The forecasting data may follow some first principles and thus lead to an equation. Our integral equation is general enough to cover a wide range of such principles, and the kernel is set to be learnable so that KatFDE can capture them accurately. In this context, KatFDE is expected to perform better; ii) KatFDE naturally inherits temporal relations from the integral equation and incorporates the attention mechanism for temporal modeling. Both are well suited for time series data with long-range dependencies, which are hard to model under LSTM frameworks; iii) KatFDE is more flexible and compatible with recent efficient attention mechanisms [4].

[1]Weng Wenchao, et al. A decomposition dynamic graph convolutional recurrent network for traffic forecasting[J]. Pattern Recognition, 2023, 142: 109670.

[2]Zhang Tianya. Network Level Spatial Temporal Traffic Forecasting with Hierarchical Attention LSTM (HierAttnLSTM)

[3]A. Vaswani, et al. Attention is all you need, NeurlPS2017.

[4]Zhou Haoyi, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI2021.

I. Novelty, Clarity, and Writing Style

1. Our key contribution is using a flexible kernel-attention fractional neural ODE that replaces fixed power-law memory kernels in traditional fractional differential equations. This framework effectively unifies and extends integer-order, variable-order, and $\psi$-Caputo dynamics within a single continuous-depth framework. Furthermore, we provide rigorous theoretical foundations for our framework, including boundedness under both singular and nonsingular cases. We establish the well-posedness of the resulting neural integral equations using Banach fixed-point arguments, ensuring solution uniqueness under appropriate conditions. The scaled dot-product attention mechanism itself is well known, and we do not claim novelty in its formulation, but showing its effectiveness as a learnable kernel in fractional derivatives is new and one of our contributions.

2. Writing Style: Our paper aims to bridge the mathematics and machine learning communities, where some readers may be unfamiliar with fractional definitions. We therefore introduce the concepts progressively, starting with the simple first-order definition, then moving to the constant-order Caputo definition, and finally to the variable- and $\psi$-order definitions. We explicitly outline these derivatives because their comparison naturally motivates our general KatFDE formulation, as discussed prior to Eq. (15).

3. We will bring the related work into the main paper if it is accepted, since there is allowance for one more page. Thank you for your suggestion.

---

II. New Supporting Systems with Experimental Results

We appreciate the reviewer’s critique of the experimental evaluation and thank them for motivating us to test our framework on more complex, non-linear, temporally evolving dynamical systems. The marginal advantage reported in the paper draft may be attributable to the evaluated datasets not being sufficiently challenging to fully demonstrate the strengths of our framework.

To further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.

1. Fluid Flow Prediction:

We evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution. Turbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in Table R1, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches.

Table R1: Average prediction error of different models on turbulent vector field prediction.

[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.

2. Urban Population Mobility Prediction:

Post-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2]. This dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in Table R2, KatFDE achieves state-of-the-art performance. This is a clear use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns.

Table R1: Average prediction error of different models on post-disaster urban mobility.

[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.

3. Biological Neural Spike Train Dynamics:

We next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context. We tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification. As Table R3 shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems. Due to limited time during the rebuttal period, we will include more baselines in the final version.

Table R3: Test Accuracy of different models on neuron spike train classification (%)

[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.

---

III. Clarify Ablation Study:

We emphasize that the baselines in Table 1 in the paper already provide a clear ablation analysis. This is fully consistent with our explanation of the novelty, which is framed relative to the first-order derivative, the constant-order Caputo derivative, and the variable-order derivative definitions. Each of these progressively introduces greater advantages and flexibility to the continuous system. Our framework fully generalizes and unifies all of these approaches, and shows clear advantages.

The ablation study is shown in Table R4, reproduced from Table 1 in the paper.

Table R4: Ablation study among differential-equation-based continuous models. The results show that through generalization, our model achieves an advantage.

Thank you for your careful review of our paper. We are grateful for your recognition of our novelty.

Furthermore, we would be honored if you could lend your insights during the discussion phase. Your perspective would be invaluable in clarifying any potential misunderstandings about our work and in contributing to a more robust discussion.

Testing Other Spatio-temporally Continuous Non-linear Temporally Evolving Systems

We thank the reviewer for this suggestion. To further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.

1. Fluid Flow Prediction:

We evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution. Turbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The complex temporal evolution of turbulence makes it an ideal test case for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in Table R1, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear complexity inherent in turbulent flows, outperforming other approaches.

Table R1: Average prediction error of different models on turbulent vector field prediction.

[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.

2. Urban Population Mobility Prediction:

Post-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2]. This dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in Table R2, KatFDE achieves state-of-the-art performance. This is a clear use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. The model can down-weight pre-disaster commuting patterns during the hurricane and re-integrate them during recovery.

Table R1: Average prediction error of different models on post-disaster urban mobility.

[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.

3. Biological Neural Spike Train Dynamics:

We next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context. We tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification. As Table R3 shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems.

Table R3: Test Accuracy of different models on neuron spike train classification (%)

[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.

Limitations Discussion

We will include the following Limitations Discussion in the paper conclusion part:

There are some limitations of KatFDE. First, KatFDE is based on a deterministic equation. In practice, we often need to handle stochastic data and outputs with confidence intervals, which the current framework does not support. Second, the spatial interaction (i.e., the spatial partial differential components) at a given time is not explicitly included in the framework with direct incorporation of physical laws. In this paper, we rely on other modules, such as GNN and CNN, to handle spatial interactions. A more natural and deeper fusion of these components may be explored in future work. Third, KatFDE may require higher computational cost when using nonlocal kernels. To make KatFDE more generally applicable, future work should explicitly include spatial interactions and consider stochastic models instead of a purely deterministic formulation. In addition, designing more efficient numerical solvers and incorporating more efficient attention mechanisms will help reduce computation and memory usage, especially for large-scale datasets.