PDETime: Rethinking Long-term Multivariate Time Series Forecasting from the Perspective of Partial Differential Equations

The literature review setion seems a little over-compressed, especially the multivariate time series literature. In fact, there is much more literature in the past 2 years that seeks to provide forecast from multiple perspectives. For example, Transformer-based methods, linear interpolation methods (D-linear) and auxiliary series construction (CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables, ICML 2024) and so on. I would suggest that the author should enrich their literature on multivariate time series, and compress the neural PDE literature because it is only utilized as a component of the model.

Also the benchmark methods are not sufficient as there are many papers in 2024 that address LMTF from LLM perspectiuve, including LLM4TS, GPT4TS, CATS and so on. I strongly encourage the author to add these benchmarks into their comparison.

I would gladly increase my rating if my concerns are addressed.

• Please write a clear problem formulation: In Time-Series forecasting, what do you have given, what do you want to predict/which objective do you want to optimize? What are the domains your inputs and outputs live in. I had to somehow guess here sometimes at the beginning, especially it was not clear to me at the beginning what $s$ is, and whether it is given or not.
• I really like the PDE motivation, but what you at the end do is simply to learn first derivations at time points instead of learning the values them self. As your spatial input $s$ is computed in the encoder and you are only considering the derivation with respect to time for fixed s, one could argue that at the end you are only considering ODEs with respect to time. However, then what you do is already established in the literature for time-series forecasting with irregular time-points, see [1], [2], [3].

[1] De Brouwer, E., Simm, J., Arany, A., & Moreau, Y. (2019). GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32.

[2]Scholz, Randolf, et al. "Latent Linear ODEs with Neural Kalman Filtering for Irregular Time Series Forecasting." (2023).

[3] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. PMLR, 2022.

• As your approach is fundamentally different to established transformer models, it is important to integrate a runtime-study. What does the performance gain over PatchTST cost with respect to efficiency?

W1. Writing Quality:

1. The paper contains undefined variables when first introduced. E.g. $\mathbf{X}{his}$, $\mathbf{c}{t}$, and $\tau_t$ in line 8, as well as $\mathbf{W}{\tau}$, $\mathbf{W}c$, $\mathbf{W}x$, $\mathbf{W}$, $\mathbf{b}\tau$, $\mathbf{b}c$, $\mathbf{b}x$, and $\mathbf{b}$. Please define any variable when it is first introduced.
2. Section 3.1 has an incorrect sub-section title. The section is titled "Problem Formulation" but describes the model's overview. Problem formulation typically refers to the specific problem the paper addresses.
3. Some statements are overly vague. For instance, in lines 502-505, "Finally, our approach of rethinking long-term ... promising direction for future research" lacks clarity. Please mention what new perspectives do the authors wish to explore?
4. Please review the order of all tables in the paper; Table 1 should appear before Table 3.
5. Typos:

5.1 line 52: hiders $\rightarrow$ hinders

5.2 line 294: $**x**{x_0} \rightarrow **x**{t_0}$

5.3 line 151: $**u**(t + \Delta) \rightarrow **u**(t + \Delta t)$

5.4 lines 77-80: "Furthermore, as shown in .... capture temporal dependencies" first sentence doesn't complete the thought

5.5 Eq. 10, $\mathcal{L}($: "(" did not close

5.6 Alg 2, line 4: $t'$ appeared twice

6. There are many other notational issues which require careful attention from authors. ex. time index have subscript at some places like eq. 2 and superscript in eq. 1

W2. Motivation for considering multivariate time series as an instance of PDE is missing. Can authors provide clear motivation behind this approach? Would this assumption hold for multivariate time series data with variables measuring different aspects, such as a physiological dataset where one variable tracks pulse and another tracks blood pressure?

W3. Model Scope:

1. Why is the model limited to long-horizon predictions? How does it perform on short horizons?
2. The authors state that existing models rely on either historical observations or time points alone. However, the Informer model uses both local and global time information as embeddings. Can authors clarify this?

W3. Encoder Architecture:

1. What is the motivation behind using this particular encoder architecture?
2. The encoder’s design is challenging to understand due to insufficient explanations:

• Clearly define and distinguish between $X^{(k)^i}$ and $\mathbf{X}^{(K)^i}$
• Provide a detailed explanation of the operations in Equation 7
• Explain the rationale behind adding $\mathbf{c}_t^{(k)}$ again after concatenation
• Explain the discrepancy between Alg 1 and Eq 7 (see line 3, Alg 1)

3. Eq. 6; why (6a) and (6b) LHS have subscript $t$ and no information of it on RHS?
4. What is For loop in Alg 1? Is it clearly mentioned in main text?

W4. Solver:

1. The paper does not clearly explain how the proposed solver circumvents the issues associated with the Euler solver. Please provide a comparison between their proposed solver and the Euler/Neural-ODE solver
2. Figure 2b does not effectively illustrate the solver. A clearer figure would help.
3. The PDE solver is claimed to be a novel and key contribution. To substantiate this, could the authors demonstrate the advantages of this solver over existing Neural ODE solvers (e.g., Chen et al., 2018) through a toy example?
4. In line 270, the authors state that Figure 3 illustrates the solver’s advantages over the Euler solver. However, it is difficult to see this comparison, as Figure 3 primarily shows results for hyperparameters $k$, $N$, and $S$.

W5. Experiments:

1. Please clarify the input and prediction sequence lengths for the results in Table 1. Comparing Tables 1 and 7, it appears the forecasting horizon is 336 (please correct me if this is incorrect)
2. There is a discrepancy in the third decimal place of Patch TST results between these tables if the forecasting horizon is 336
3. Please inform about Table 7 in paragraph after Section 4.2, otherwise the text is confusing

1. The use of numerical solvers (e.g., Euler Solver) could introduce sensitivity to hyper-parameter tuning and potential limitations in modeling high-frequency or abrupt temporal changes.
2. The encoding-integration-decoding process, along with meta-optimization, may add computational overhead, especially for large datasets or models with high-dimensional data, limiting scalability.

1. Fairness in Comparison: My most concern is that the experiments may lack fairness due to differences in historical input length ($H$) between PDETime and baseline models. While PDETime’s $H$ is optimized, baseline models use a fixed input length, which can skew results since (1) different input lengths impact the number of samples in the test set for each model, potentially affecting comparability, and (2) input length significantly influences forecasting performance. It’s recommended to either standardize $H$ across models or optimize it for all baselines.
2. Code Availability: The absence of released code reduces the credibility and reproducibility of the results, as reviewers and readers cannot verify the findings independently.
3. Unclear Loss Weighting: The weight description for the loss function in Equation 14 is unclear, making it challenging to understand the balance between different loss components.
4. Lack of Efficiency Analysis: There is no discussion of PDETime’s computational efficiency, such as runtime or memory consumption, which is essential for understanding its scalability and practicality for long-term forecasting tasks.