VMFTransformer: An Angle-Preserving and Auto-Scaling Machine for Multi-horizon Probabilistic Forecasting

Thanks for your thoughtful feedback.

W1: We appreciate the reviewer raising this important point about clearly explaining the intuition and implementation details behind the proposed "Angle&Scale" similarity measure.

The key motivation for Angle&Scale similarity is to better match the structure of our proposed VMF-Transformer objective function, which models the multi-horizon forecast target as a random vector characterized by both an angle/direction and a magnitude/scale. As explained in Section 3.1, the VMF distribution captures the dependence structure across horizons through the angle or direction of the target vector. The scale of the vector represents the magnitudes of the target at each timestep.

Accordingly, the Angle&Scale similarity is designed to evaluate relevance based on two facets - angle and scale. For two vectors in the attention module, we first compute the cosine of their angle to assess similarity in orientation. We additionally multiply this by a Gaussian kernel applied on the difference in vector lengths. This accounts for whether two vectors have a similar scale or magnitude.

Intuitively, this allows the model to match historical sequences in the attention module that have both a similar direction and scale profile to the forecast target. We empirically demonstrate in Section 4 that using Angle&Scale similarity outperforms baseline dot product attention across multiple metrics and datasets. The results suggest that incorporating both angle and scale information is beneficial for this forecasting approach.

The shape of $Z$ is $(t+1)\times (H*d+1)$, where $t+1$ is the length of the history (we allow the index of the time series sequence to start from 0 instead of 1.). The $i$-th row of $Z$ is a vector $[y_i \circ \mathbf{x}^T_{i+1}\circ \cdots \circ \mathbf{x}^T_{i+H}]$, where $y_i$ is the ground truth at time $i$, and $\mathbf{x}^T_{i+1}, \cdots \mathbf{x}^T_{i+H}$ are features or known inputs corresponding to the future. Each $\mathbf{x}_{i+1}$ is a d-dimension vector, and they are packed into one row vector by simply transpose and concatenation together with the real value $y_i$ . Since the prediction length is $H$, there are a total number of $H$ input vectors $\mathbf{x}$. Hence the length of each row vector of $Z$ is $H\times d+1$. And therefore, $Z$ is a $(t+1)\times (H \* d+1)$ matrix. For $Q$ ,$K$, and $V$, they are all originated from $Z$ by applying different convolutional kernels to $Z$, where the convolutional kernels are learned during the training stage. The kernels are all 1D, and hence do not change the 'length' of $Z$, but compress the 'width' of $Z$ from $(H \* d+1)$ to $d_q, d_k$ and $d_v$ for $Q$ ,$K$, and $V$ respectively.

W2, W3, and W4: we thank you for your advice. The revised draft is presented in general responses (from part 1 to part 5).

We thank the reviewer for offering valuable feedback. We have addressed each of the concerns raised by the reviewer as outlined below.

1. The reviewers rightly point out that many time series exhibit strong seasonal patterns that can dominate the signal. We analyzed the strength of the seasonal component in the electricity, solar, and traffic datasets using autocorrelation analysis. The electricity data shows a strong daily seasonal pattern with autocorrelation remaining above 0.6 at the 24 hour lag. The solar data shows both daily and yearly seasonal patterns due to the underlying physical processes. The traffic data has relatively weak seasonality. However, basic time series methods such as auto.arima and exponential smoothing often cannot effectively model complex real-world time series. ARIMA, exponential smoothing, and Prophet on our used datasets achieve relatively poor performance compared to more flexible machine learning approaches [1, 2]. These methods make strong assumptions about the underlying data generative processes and do not directly capture uncertainties. Our model demonstrates excellent performance across datasets with varying degrees of seasonality. By jointly modeling the sequential dependence, overall scale, and capturing uncertainty, our method provides robust performance without needing an explicit seasonal component model. We provide quantitative metrics showing our model's ability to accurately forecast across a diverse range of conditions.

[1] Lim, Bryan, et al. "Temporal fusion transformers for interpretable multi-horizon time series forecasting." International Journal of Forecasting 37.4 (2021): 1748-1764.

[2] Wang, Y., Smola, A., Maddix, D., Gasthaus, J., Foster, D., & Januschowski, T. (2019, May). Deep factors for forecasting. In International conference on machine learning (pp. 6607-6617). PMLR.

2. Learning curves (Figure 3) of our model are provided in the updated version to demonstrate the stability of the training procedure.

3. A comparison of metrics corresponding to different horizons where H ranges from 1 to 720 (Figure 4) of our model is provided in the updated version to demonstrate the stability of forecasting performance.

4. This revised draft incorporates extensive additional experiments in general responses (from part 1 to part 5), now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

5. We actually used the negative log-likelihood (NLL) in the training stage. As we explained in our paper ' The training object is to maximize Equation 6 with respect to $\Theta$, or equivalently, minimize the inverse $-l(\Theta)$', where $-l(\Theta)$ refers to the NLL. Thanks for noticing. We now realize this may be a confusing statement and we will clarify it to enhance readability and comprehension.

Thank you for taking the time to review our paper and provide constructive feedback. We appreciate the reviewers recognizing the potential of our proposed VMFTransformer model for multi-horizon probabilistic time series forecasting.

W1, W2, and W5: this revised draft incorporates extensive additional experiments in general responses (from part 1 to part 5), now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

W3: We will revise the formatting of all equations in the paper to enhance readability and comprehension. Equations are centered on their own lines with consistent spacing around operators. Variable names have been standardized and vector/matrix notation clarified. We believe these changes significantly improve the formatting.

W4: The main contribution of our work is the novel probabilistic forecasting model based on the von Mises-Fisher distribution and Transformer architecture. To better articulate this, we will add an explicit contribution statement in the introduction highlighting:

1. The VMFTransformer model itself which captures temporal dependence in a multi-horizon target.
2. The Angle&Scale similarity measure for the self-attention module
3. An efficient method to optimize the likelihood function with the Bessel function

Additionally, we will reorganize several sections to improve logical flow and aid comprehension of the technical details underpinning the model innovation. A graphical abstract will also be added to provide an intuitive overview. We believe these changes address the need to better communicate the main contributions of our work.

Dear Authors,

Thank you for providing additional experimental results in your rebuttal. I appreciate your efforts to address the concerns raised in the initial review. However, after careful consideration, I believe there is still room for improvement, particularly in terms of the experimental setup and the consistency of results.

One key issue is that your paper does not align with the standard experimental setups used in other works within the literature, especially regarding the forecasting horizon. Notably, your results for a 720-horizon forecast appear to be inconsistent with those reported by the original authors in the following two referenced papers. To facilitate a more straightforward and meaningful comparison, I suggest that you consider adjusting your experimental setup to adhere to the standard benchmarks prevalent in the literature. This would not only enhance the clarity of your results but also their comparability with existing studies.

I have included references to two pertinent papers below for your consideration. These papers should serve as a guide for the standard numerical values and methodologies typically employed in this field of research.

Given these considerations, I have decided to maintain my original review score. I believe that addressing these concerns will significantly strengthen the impact and relevance of your work in the context of existing literature.

Looking forward to seeing the revised version of your manuscript.

Best regards, Link to github repo of the papers. https://github.com/cure-lab/LTSF-Linear

https://github.com/yuqinie98/PatchTST

We deeply appreciate the thoughtful feedback provided by the reviewers. Their suggestions have helped strengthen our work substantially. In response:

W1 and W4: Per the reviewers' recommendations, we have added extensive additional experiments with the latest baseline models across longer time horizons. Every key result now includes the mean and standard deviation to rigorously quantify uncertainty. We believe these comprehensive additions showcase our method's strengths.

W2: You raises an important point regarding the motivation behind using the von Mises-Fisher (VMF) distribution and truncated normal distribution to model the multi-horizon forecasting vector. The key intuition is that the VMF distribution can effectively characterize vectors distributed around a specific direction, which fits our assumption that when dependence exists among the multi-horizon targets, they will concentrate around a certain direction. The truncated normal distribution provides a flexible prior for the scale parameter to make the VMF likelihood tractable. Compared to simply using independent normal or Laplace distributions on each horizon, the VMF and truncated normal distribution can jointly capture the dependence structure via the shared direction parameter in VMF. We will add more discussion on the motivation in the revised paper to clarify this modeling choice.

W3: We carefully re-evaluated TransMAF, but unfortunately found it performed poorly on our datasets - significantly worse than the other baseline models we assessed. As such, we decided not to include these inferior results and instead added evaluations using the popular PatchTST and DLinear models, providing readers with a more representative view of the state-of-the-art. Regarding evaluation metrics, we utilize the community standards of MAE, MSE, Q50-Loss, and Q90-Loss - prevalent metrics used across probabilistic time series works.

We thank the reviewers again for their outstanding suggestions. We believe the additions made in response (from part 1 to part 5 in general response) further demonstrate the competitiveness and rigor of our proposed approach across both model architectures.