Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting

Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.

Re: Claims And Evidence & Experimental Designs Or Analyses & W1 & W2 & C2 & Q1

Due to space limits, please refer to Re: W1 in the Rebuttal for Reviewer ZfWm.

Re: Methods And Evaluation Criteria & Supplementary Material & W3 & Q2

• KD's role in enhancing feature diversity:

Due to space limits, please refer to Re: W2 in the Rebuttal for Reviewer ZfWm.

• KD's impact on robustness:

We observed performance gains by incorporating the KD module. The reason behind could be that KD dynamically tunes the balance across different time series, enabling MLPs with limited capacity to favor information containing less noise. However, the improvement in predictive performance is less prominent compared to robustness enhancements brought by the RSTIB, because in all situations, we still need to balance the preservation of the target and the compression of all the reparameterization. Intrinsically, although KD can tune the relative ratio, the improvement regarding predictive performance is still constrained by the objective itself.

• The impact of different regularization terms:

We provided further ablation studies and discussions on the respective roles of the regularization terms in Appendix K.2. Please refer to that section for more details.

Re: Relation To Broader Scientific Literature

Appendix I.2 discusses the distinctions between RSTIB/RSTIB-MLP and existing IB methods. Please refer to that section for more details.

Re: Essential References Not Discussed

We highly appreciate the suggested papers! Below we provide a detailed discussion on the relation between our submission and the cited works:

• We are inspired that we could include an additional part for introducing the robust spatial-temporal forecasting: RobustTSF ([1]) considers time series forecasting with anomalies, while our work similarly considers spatial-temporal forecasting with noise perturbation. We share a similar way of experimental conduction: artificially introducing noise and data missing (specifically mentioned in Essential References Not Discussed). Including this paper can enhance our experimental settings and results' credibility.

• Linking feature diversity with spatiotemporal heterogeneity from [2] is really intriguing. Both works consider robustness in spatial-temporal forecasting. Our work attempts to enhance the feature diversity, while [2] aims to capture the heterogeneity. Our work quantifies feature diversity using feature variance(Var), [2] quantifies heterogeneity using the Average Node Deviation (AND) metric:

Given the feature matrix $X \in \mathbb{R}^{n \times d}$, the AND metric is defined as:

$$D(X) = \frac{1}{n^2}\sum_{i=1}^n \sum_{j=1}^n \sum_{k=1}^d (x_{ik}-x_{jk})^2$$

To link it with Var, we further derive $D(X)$ as below:

\begin{aligned} D(X) &= \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \sum_{k=1}^d (x_{ik} - x_{jk})^2 \\ &= \frac{1}{n^2} \sum_{i,j,k} \left( x_{ik}^2 + x_{jk}^2 - 2x_{ik}x_{jk} \right) \\ &= \frac{1}{n^2} \left( 2n \sum_{k,i} x_{ik}^2 - 2 \sum_{k}\left( \sum_{i} x_{ik} \right)\left( \sum_{j} x_{jk} \right) \right) \\ &= \frac{2}{n^2} \left( n \sum_{k,i} x_{ik}^2 - n^2 \sum_{k} \bar{x}_k^2 \right) \\ &= \frac{2(n-1)}{n} \sum _ {k=1}^d \left( \frac{1}{n-1} \left( \sum _ {i=1}^n x _ {ik}^2 - n\bar{x} _ k^2 \right) \right) \\ &= \frac{2(n-1)}{n} \cdot \text{tr}(\mathbf{Cov}), \end{aligned}

where $\bar{x} _ k = \frac{1}{n} \sum _ {i=1}^n x _ {ik}$ and $\text{tr}(\mathbf{Cov}) = \sum_{k=1}^d \frac{1}{n-1} \left( \sum_{i=1}^n x_{ik}^2 - n\bar{x}_k^2 \right)$.

Thus, the AND metric is proportional to the trace of the covariance matrix $\mathbf{Cov}$:

$$D(X) \propto \text{tr}(\mathbf{Cov})$$

The AND metric emphasizes overall variance, whereas our Var metric emphasizes balanced standard deviations.

We'll extend the above in our final version.

Re: C1

Many thanks for this constructive advice! We suspect that the mentioned "extreme noise conditions" refer to real-world noise experiments (as mentioned in C2), such as data-missing scenarios. We have actually provided such evaluations and discussions in Appendix K.3. Your suggestion is helpful as it guides us towards more focused and clear settings. Nevertheless, by including robustness studies under diverse noise conditions, we have also demonstrated the consistent performance of our method.

We hope this addresses all your concerns. Thank you very much!

Re: Supplementary Material

We are sincerely sorry for the typos, and will correct them in our final version.

P16: "illustrated in Fig.7(a)"; "represented in Fig.7(b)".

Re: Essential References Not Discussed

Appendix K.7 has examined such scenarios where transformer-based models with a very large number of parameters (PDFormerand STAEformer) are compared.

Please note that achieving new SOTA performance is not our claim. Instead, we dedicate to achieving a good trade-off between robustness and efficiency, which is demonstrated with SOTA STGNNs ((iii) above Related Work). We will include and discuss these three papers and further clarify this point in our final version.

Re: W1

It is evaluated theoretically (Appendix H) and empirically (Section 5.3).

Below we additionally evaluate the overall training to convergence time (TTCT) in PEMS04 and the training time per epoch(TTPE) on the large Weather2K-R dataset:

It is clear that our full training is much faster; e.g., RSTIB-MLP reduces up to 88.42% compared to STExplainer. Besides, the superior efficiency of RSTIB-MLP is consistent on the large dataset.

These results will be included to ensure comprehensiveness.

Re: W2

(i) Please refer to Re: W1 in the Rebuttal for Reviewer ZfWm.

(ii) We follow the setting of RGIB as follows:

• The train set is added with input noise and target noise.
• The validate and test sets are added with the same set of input noise, while their targets remain clean (unchanged) to ensure accurate evaluation.

All methods follow these settings.

Re: W3 & Q3

Yes! we'll leave rooms in the main text for below:

• Assumption 4.1:

The sliding window mechanism is a technique for processing ST data, extracting fixed-length subsequences from raw data by progressively sliding a window over temporal or spatial-temporal dimensions. A critical feature is that the same data window can flexibly serve as either input or target, creating a "dual noise effect"—when a noisy sequence serves as both the input $X$ in one window and the target $Y$ in another, noise propagates bidirectionally. If $Z - X - Y$ holds, then $I(Z; Y|X) = 0$: The noisy information behind $I(Z; Y|X)$ is directly ignored. Ignoring noise in $Y$ is therefore problematic. The dual noise effect allows noise influences both input and target across overlapping windows, necessitating relaxation of $Z - X - Y$.

• Assumption 4.2:

ST graphs exhibit invariant patterns (generalizable across time) and variant patterns (node-specific, time-varying dynamics). Invariant patterns might represent structural dependencies (e.g., road connectivity in traffic prediction), whereas variant patterns could reflect transient events (e.g., traffic congestion due to accidents). Data dynamics thus also depend on the current window's characteristics, meaning the prediction for $Y$ is not entirely determined by $X$, but also by $Y$'s unique dynamics. Therefore, the assumption $Z - X - Y$ requires relaxation.

Re: Q1

Please refer to Re: Experimental Designs or Analyses in the rebuttal for Reviewer WNiV.

Re: Q2

To achieve this, a method calculating the noise impact indicators without supervised signals is required. Below provides one possible alternative unsupervised solution to our KD approach:

Firstly, perturb the noisy $X$ with noise to generate $X_{\text{perturb}}$. Then:

$$\hat{\alpha_i} = \frac{\exp\left(D\left(X_{\text{perturb}, i}, X\right)\right)}{\sum_{j=1}^{N} \exp\left(D\left(X_{\text{perturb}, j}, X_j\right)\right)}, \quad \forall i \in \{1, \ldots, N\}$$

The following training process is similar to RSTIB-MLP with (w/) KD.

We investigate perturbation ratios of 0.1, 0.3, 0.5, and noise ratios of 0.1, 0.3, 0.5 to compare RSTIB-MLP with KD. Best results for the unsupervised method (RSTIB-MLP w/ Aug) across all perturbation ratios are reported below:

Results demonstrate that RSTIB-MLP w/ Aug cannot outperform RSTIB-MLP w/ KD. Potential reasons(risks):

• Noise ratio is difficult to characterize precisely. Fixed perturbation ratios and augmented $X_{\text{perturb}}$ samples may not align with real scenarios. Even adjustable ratios still pose this risk.

• Using only information from $X$ neglects dynamics unique to $Y$ (Assumption 4.2). Calculating the noise impact indicators solely from $X$ thus yields relatively inaccurate quantification.

This underscores the importance of our KD approach.

We hope this addresses all your concerns. Thank you very much!

Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.

Re: W1

Please note that the derivation of the RSTIB principle and the implementation of RSTIB-MLP do not make assumptions regarding the noise type. AWGN is employed in our experiments primarily because it is commonly adopted as a noise model in information theory and experimental measurements within time series forecasting ([Ref1]).

To better address this concern, we have provided further empirical study in Appendix K.3. This section details how RSTIB-MLP handles data missing scenarios compared with selected baselines, and also the contribution of each module under such noisy conditions.

Re: W2

In our settings, the Lagrange multipliers are set to be constants. This means that the balance between preserving the target and applying regularization is fixed across different time series, which may not be optimal. Besides, regularization further limits the capacity of the MLP, preventing the model from fully learning complex features in the data. Therefore, we aim to better balance the preservation of the target and the regularization terms across different time series.

Inspired by the observation that the feature variance will largely decrease as the noise ratio increases, we leverage the KD module based on the noise impact indicator information to achieve what has been described in the Training Regime: When noise impact is low, we relax the KD-based regularization; when there is a significant noise impact, we intensify the KD-based regularization. As a result, the feature variance can be increased by KD in the cases when the noise impact indicator shows high noise impact. Please note that noise impact indicators quantify the noise impact on each time series and this noise impact information is used to dynamically tune/adjust the optimization of our model.

Hope that we have addressed all your concerns. Thank you very much!

[Ref1] Teck Por Lim and Sadasivan Puthusserypady. Chaotic time series prediction and additive white Gaussian noise. Physics Letters A, 365(4):309–314, 2007.

Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.

Re: Experimental Designs Or Analyses

• "What are the teacher model(s) used in the experiments in Table 2?"

Teacher model selection settings are described below Figure 4: "Our method is teacher model agnostic (Appendix. K.10), where we set the default teacher model to STGCN."

• "How will the results differ if other teacher models were used?"

This is discussed in Appendix K.10, where we directly compare empirical results obtained using different teacher models. Table 18 provides empirical details, and the process of preparing the teacher model is also described in Appendix K.10. We demonstrate that the superior performance of RSTIB-MLP is independent of the teacher model choice by showing that RSTIB-MLP also achieves strong robustness even when selecting MLP as the teacher model—outperforming or performing comparably to some state-of-the-art (SOTA) STGNNs. The potential reason behind this is elaborated in Appendix K.10. We will clarify this further in our final version.

• "What is the performance of the teacher model before distillation?"

Consistent with the previous response, we present an empirical study in Appendix K.10 to demonstrate that RSTIB-MLP’s performance is independent of the original teacher model's performance. Even when using a basic MLP as the teacher model, RSTIB-MLP can achieve better or comparably good robustness relative to some SOTA STGNNs.

• "Was this a distillation using only the output of the network, or were intermediate features used?"

Only the output of the network is utilized. Specifically, Definition 4.9 clarifies that the teacher model's output is used exclusively to compute the noise impact indicator.

Re: Other Strengths And Weaknesses

• "What loss function is used in the MLPs?"

Below is our loss function:

$$\mathcal{L}_{RSTIB\text{-}MLP} = \sum _ {i=1}^{N}\left[-\mathcal{L} _ {\text{reg}}(Y_i^S, \tilde{Y}_i)\right] + \sum _ {i=1}^{N}(1 + \hat{\alpha} _ i)(\lambda _ x \mathcal{L} _ {x,i} + \lambda _ y \mathcal{L} _ {y,i} + \lambda _ z \mathcal{L} _ {z,i})$$

This is also described in Eq.(6). Specifically, $\mathcal{L} _ {\text{reg}}(Y _ i^S, \tilde{Y} _ i)$ is the lower bound of $I(Z;\tilde{Y})$. Please refer to Proposition 4.8 for implementation details and proofs. Additionally, $\mathcal{L} _ {x,i}$, $\mathcal{L} _ {y,i}$, and $\mathcal{L} _ {z,i}$ represent the corresponding regularization terms applied to the input, target, and representation regions respectively. For their analytical calculations, please refer to Proposition 4.6 and Proposition 4.7. Descriptions of how we implement these regularizations are provided above Proposition 4.6 (for input and target) and Proposition 4.7 (for representation regularization). Moreover, $\hat{\alpha}$ is the noise impact indicator defined in Definition 4.9, serving to dynamically adjust the training of RSTIB-MLP when incorporating these regularization techniques.

• "Were the teacher models pretrained or trained from scratch? In general, more details on the distillation procedure should be given."

Relevant details are included in Appendix K.10 due to space limitations. We apologize for the unclear statement in the main text. The teacher model is trained from scratch by ourselves. The knowledge distillation procedure is implemented as follows:

1). Train the teacher model from scratch, following the same procedure as the student model.

2). Freeze the teacher model's parameters and start training RSTIB-MLP.

3). Only the output of the teacher model is leveraged to calculate the noise impact indicator as defined in Definition 4.9.

According to Definition 4.9, the noise impact indicator is normalized using the Softmax function, reflecting the relative relationships among time series.

Re: Other Comments Or Suggestions

Many thanks and we really appreciate for the suggested idea! Yes, the efficiency is one key advantage that we aim to demonstrate. Therefore, we include "MLPs" in our title to stress the point of our method. We will consider this constructive suggestion in our final version.

We hope we have addressed all your concerns. Thank you very much!