Stochastically Capturing Partial Relationship among Features for Multivariate Forecasting

1. This paper lacks a detailed rationale for why randomly selecting feature subsets from multivariate time series can enhance forecasting performance. Specifically, if the training and testing sets consist of different feature subsets, it is unclear how this method ensures good generalization performance across unseen data.
2. Although the authors have limited their analysis to non-informative distributions, many real-world time series datasets possess inherent hierarchical structures (e.g., cash flow, population data). Applying random permutation operations in such cases may disrupt these intrinsic relationships, potentially leading to suboptimal performance.
3. The theoretical analysis does not sufficiently explain why random feature subsampling would improve generalization performance.

The mathematical notations are not clear enough, which reduces the readability of the paper and the overall score of this paper. Below are some questions on the notations:

1. It is very confusing to write $\bf{F} \sim \mathcal{P}(\bf{F})$ in Eqn. (1). Is $\bf{F}$ a subset or a probability value when inputting to $f$ to get the estimated value?
2. What is the dimension of $\bf{h} _ {b,i}^{(0)}$? How do one concatenate $\bf{h} _ {b,i}^{(0)}$ to form $\bf{h}^{(0)}$?
3. What are the dimensions of $\bf{e} _ {b}^{Time}$ and $\bf{e} _ {\bf{F} _ i}^{Feature}$ in Eqn. (2)? It seems that the vectors $\bf{e} _ {b}^{Time}$ and $\bf{e} _ {\bf{F} _ i}^{Feature}$ are different and addition is impossible. Further, how do one concatenate $\bf{e} _ {b}^{Time}$ ($\bf{e} _ {\bf{F} _ i}^{Feature}$) to form $\bf{e}^{Time}$ ($\bf{e}^{Feature}$)?
4. Is Eqn. (7) valid for any arbitrary loss functions $l$? Please justify in the paper.

In order of gravity:

1. There are two flaws in the provided theoretical results:

• (major) Theorem 1, as it is derived from the PAC-Bayesian classical results of [McAllester 1991], does not hold for time-series data, because their training sets are not I.I.D. . Indeed the bound stands in probability over independent draws of samples of m instances, while time-series instances notoriously show time dependencies.
• (minor) The conjecture of Section 3.4 about why results improve with the number of inference runs $N_I$ is not supported by the reasoning of lines 241-244 because the derived probability of picking a subset increases not only for highly correlated subsets of features, but for any subset of features, including subsets of unrelated features which, if drawn more often, should result in decreased results, as argued in lines 49-50.

2. Standard errors are not reported for any result. Without them, it's impossible to judge the significance of the improvements, hence to support the claim of e.g., lines 23-25, that "SPMformer outperforms baselines".

3. The comparison between methods based on FLOPs reported in Figure 7 is not fair, because it accounts only for the number of operations needed to run a forward pass for a single subset of features (line 453). Because SPMformer requires running the model several times (number of subsets times number of trials), a fair comparison would account for the overall operations required for a full inference pass.

4. The paper does not mention how the hyper-parameters have been selected.

5. The 0.5-risk is not used in the papers [Zhou 2021, Makridakis 2022] as mentioned in line 315.

1. The motivation for the proposed method has unclear points. In the introduction section, the authors mention that "methods often face limitations due to their deterministic approach to grouping." The deterministic approach is not necessarily limited in this regard if it manages to capture the underlying stable pattern in data. Even if this is the case, it is incomprehensible how randomly sampling variable subsets can solve the issue, and meanwhile, without principled sampling and an understanding of data patterns, the proposed randomized method introduces additional issues like missing important variables and ignoring inter-relations across variables. The example of stock is opposite to the motivation, because sectors and trading regions/markets of stocks are mostly static, and market caps are slowly changing.

   Essentially, the introduction does not justify the necessity and rationale of randomizing the input variables of prediction models during training and inference, nor does it recognize the additional issues introduced by randomizing input variables.

2. The proposed method (i.e., from Eq.1 to 6) is mostly applied, with little technical contribution and new insights. In Eq.1, the distribution $\mathcal{P}$ is not formulated throughout the paper and is eventually realized by a uniform distribution, making the method less principled.

3. In lines 166 to 167, it is mentioned that "However, for stochastic partial multivariate cases, F can vary when re-sampled, requiring to ability to deal with dynamic relationships". This claim is ambiguous and depends on the definition of dynamic relationships. If the dynamic relationship is at the instance level, varying input variables for training and inference can hardly handle this case. The authors should be cautious about overclaims.

4. In Eq.2, the segmentation and tokenization process seems to share the linear projection. As the input variables change and their patterns might differ significantly during training and inference, sharing the projection in Eq.2 might be suboptimal.

5. In line 197, does "the concatenated representations" lead to extremely high dimensions if it is the product of the timestep and the hidden dimension?

6. In line 7 of Algorithm 1, the averaged loss is confusing, since each element corresponds to a different set of input variables.

7. Sec. 3.4 discusses the number of samples for inference, however, what also matters for inference is the variable sample size. In the Other Settings of the experiment section, the sample size is dataset-specific, and there is no guidance on how to set it.

8. The theoretical analysis rarely answers the rationale of the method. Theorem 2, as presented in lines 270-289, primarily boils down to the impact of sample size and instance number, which is rather superficial and does not provide meaningful insight into the method’s effectiveness.