Robust Spatio-Temporal Centralized Interaction for OOD Learning

Thank you very much for your review. We provide figures and tables at an anonymous link ( https://anonymous.4open.science/r/reb_/5Pwy.pdf ). The "#" before a title means that complete tables in the section are available in the anonymous link.

# Meta-learning

We introduce meta-learning based spatiotemporal models, and results on SD dataset show that our model outperforms these models.

# IID

Using SD dataset as an example, the table below reports the average MAE. This demonstrates that our model still achieves competitive performance under the IID setting.

Computational Complexity

As demonstrated in Theorem 1, our centralized messaging has higher efficiency. While the computational complexity of node-to-node interaction is O( $N^2$ d), our method reduces it to O(PNd), where P is much smaller than N. For example, P can be as small as 8 for a graph with 716 nodes.

Large Spatiotemporal Dataset

As noted in line 309, in Section E.3, we evaluate the performance on large datasets. Below, we summarize key results from the CA dataset for easy reference. These results confirm the strong effectiveness of our model for large-scale data.

# Non-spatiotemporal Domain

We regret that, due to significant conceptual and methodological differences between the spatiotemporal domain we focus on and the reinforcement learning and sequential decision-making areas of interest to you, we cannot explore these unrelated areas within the limited time available.

To address your concerns, we extended our evaluation to discrete graph OOD learning. Following the setup in [1], the table below on Collab-0.8 dataset shows our method performs well in non-spatiotemporal domains. "w/o DRO" and "w/o Per" indicate the removal of the DRO and message perturbation mechanisms, respectively, underscoring their positive effects in non-spatiotemporal setting.

[1] Zhang Z, Wang X, et al. Out-of-distribution generalized dynamic graph neural network with disentangled intervention and invariance promotion. arXiv preprint.

Different Types of Shifts

In the third paragraph of the introduction and in Figure 1, we analyze the effects of different shifts. Temporal shifts lead to inaccurate node descriptions, with representation errors propagating through node-to-node messaging and impacting other nodes. spatial shifts disrupt established message passing paths, preventing the model from following trained pathways and degrading its graph representation performance. Appendix section E.5 evaluates our method’s ability to address these shifts. To address your concern, we will refine the third paragraph to better emphasize the distinct impacts of each shift.

Related Work

Sorry for any confusion. Due to space constraints, we discuss related works to spatiotemporal OOD learning in detail in Appendix Section A. In the new version, ICML permits an additional page, which will allow us to incorporate this discussion.

# Long-term temporal dependencies

Following the setup in long-term time series analysis, we use the Knowair dataset as an example, employing 96 time steps to predict 720 time steps. The averaged results below demonstrate that our proposed model effectively handles long-term temporal dependencies.

Decision-making Process

Although model interpretability—an active research area—is not the primary focus of this paper, which emphasizes spatiotemporal OOD generalization—we analyze the prediction decision-making process in Section E.14 using Shapley values, a standard tool for evaluating component contributions. Our model integrates temporal and spatial components to generate the final output. Under spatial shifts, the temporal component dominates, while under temporal shifts, the spatial component takes precedence. Section E.15 further improves interpretability through embedding visualization. In future, we will explore enhancing interpretability by incorporating insights from sequential decision-making or reinforcement learning.

Potential Applications

Spatiotemporal prediction, to which our method belongs, plays an active role in various downstream applications. For example, traffic forecasting can help urban planners optimize city designs, while meteorology forecasting can guide the public in understanding future climate and environmental conditions.

Presentation

We will use the full terms instead of acronyms, except for the name of the model STOP. Moreover, we will remove symbols following formulas.

Thank you sincerely for your appreciation and thoughtful comments regarding the paper. Your feedback is invaluable in enhancing the quality of the manuscript.

W1. Visualization Cases

Due to the rebuttal policy restrictions of ICML, we were only able to include some prediction visualization cases in an anonymous link (https://anonymous.4open.science/r/_reb/objW.pdf). Meanwhile, in the submitted manuscript, Figure 9 visualizes the time embeddings, personalized embeddings, and contextual embeddings used in our model, providing evidence for the effectiveness of the proposed embedding techniques.

W2. Zero-shot Performance

In fact, the zero-shot performance includes the results for newly added nodes, as shown in Table 4 of our submitted manuscript. These nodes had no observed data during training and were excluded from the training process. Despite this, our model demonstrates impressive zero-shot performance. This is largely due to its ability to extract shared contextual features from spatiotemporal data, which can be leveraged by newly added nodes to achieve effective representations. For convenience, we have included the zero-shot performance results for SD below.

W3. Few-shot Performance

To address your question, we use the SD dataset as an example. Following the experiment setup in the paper, when the testing environment changes, we fine-tune the model using the first week of the test set to evaluate its few-shot performance. The results are shown in the table below. We find that with minimal fine-tuning data, our model quickly adapts to new spatiotemporal patterns, demonstrating strong generalization. This is largely due to the lightweight architecture and effective use of contextual features.

W4. Related Work

In Section A.3, we summarize recent advances in continual learning. While these methods address dynamic spatiotemporal data, they typically require extensive new data for fine-tuning, implicitly assuming independent and identically distributed (i.i.d.) data.

In the updated paper, we discuss progress in out-of-distribution (OOD) learning. Traditional machine learning assumes training and testing data share the same distribution, but real-world applications often face distribution shifts, leading to significant performance drops post-deployment [1]. This has driven growing interest in OOD learning, with methods categorized into unsupervised representation learning, supervised model learning, and optimization-based approaches [2-3]. These techniques leverage causal and invariant learning to extract generalizable knowledge from latent test distributions.

Recently, OOD learning in graph learning has gained attention. For example, DisenGCN [4] disentangles informative factors in graph data, assigning them to distinct parts of vector representations. However, these methods struggle with spatiotemporal OOD problems due to their inability to capture complex, heterogeneous spatiotemporal correlations.

Our key contribution is a novel spatiotemporal interaction mechanism that integrates centralized message-passing, message perturbation, and a distributionally robust optimization (DRO) objective, addressing these limitations effectively.

Reference:

[1] Yang, Jingkang, et al. "Generalized out-of-distribution detection: A survey." ICCV 2024.

[2] Liu, Jiashuo, et al. "Towards out-of-distribution generalization: A survey." arXiv preprint arXiv:2108.13624 (2021).

[3] Kaddour, Jean, et al. "Causal machine learning: A survey and open problems." arXiv preprint arXiv:2206.15475 (2022).

[4] Ma, Jianxin, et al. "Disentangled graph convolutional networks." ICML 2019.

Q1. Potential Improvement

In Section F, we discuss potential improvements to the model, including enhancing its performance by integrating large language models and implementing more advanced perturbation mechanisms, among others.

Q2. GCN

Whether GCNs are essential for spatiotemporal forecasting remains debated. However, research consistently shows that GCN-based models outperform those relying solely on temporal dependencies, owing to their ability to introduce inductive biases among nodes. In general OOD scenarios, the dense message-passing mechanism of GCNs exhibits sensitivity to distributional shifts. To address this, we propose a centralized interaction mechanism as an alternative, enhancing the robustness of spatiotemporal interactions.

We deeply appreciate your valuable time and effort, which are crucial for improving the quality of our manuscript.

W1. General Graph OOD Learning Discussion

In the field of graph representation learning, researchers have focused on modifying model architectures to enhance the representation of invariant knowledge, thereby improving generalization performance for OOD problems [1-2]. For instance, GAUG [3] improves downstream training and inference by incorporating an edge prediction module to modify the input graph. DisenGCN [4] emphasizes learning disentangled representations by separating distinct and informative factors from graph data and assigning them to different parts of the decomposed vector representations. OOD-GNN [5] introduces a nonlinear graph representation decorrelation method, leveraging random Fourier features to eliminate statistical dependencies between causal and non-causal graph representations generated by the graph encoder. However, these models fall short in capturing the complex heterogeneous spatiotemporal correlations inherent in spatiotemporal data, resulting in suboptimal performance for spatiotemporal OOD tasks. In this paper, we propose a novel spatiotemporal interaction module to enhance the robustness of models against spatiotemporal shifts. We will add this discussion to the next version.

W2. More challenging conditions

In Section E.6 of the appendix, we discuss performance comparisons in scenarios where the graph topology grows rapidly. To further evaluate robustness under extreme conditions, we use the SD dataset as an example, where the size of the test graph topology is nine times larger than that of the training graph. The performance results are shown in the figure below.

W3. Comprehensive Efficiency Analysis

We sincerely apologize for any lack of clarity in our explanation. We report comprehensive efficiency metrics for several SOTA models on the SD dataset, including simultaneous training time, total training time, inference time, memory usage under optimal performance conditions, and average MAE performance. We find that our proposed method achieves competitive efficiency compared to the SOTA model D $^2$ STGNN. This is attributed to our model's reliance on a lightweight MLP architecture.

Presentation Problem

We sincerely appreciate your valuable suggestions. In the new version, we will optimize the structure of the paper and correct typographical errors to further improve the overall quality and readability of the manuscript.

Q1. MLP- vs Transformer- based Models

STID outperforms D $^2$ STGNN in some experimental results, potentially because STID leverages various embedding techniques to capture prior information, which is beneficial for enhancing the model's generalization capability. On the other hand, D2STGNN has a more complex parameter structure, making it prone to overfitting in the training environment. This overfitting can lead to a decline in its ability to generalize to unseen scenarios.

Q2. Temporal shifts

We tackle temporal shifts using two key techniques: (1) Temporal Embedding Technology : By encoding day-of-week and timestamp-of-day information, this approach captures multi-level periodic patterns in spatiotemporal data, strengthening the model's ability to represent stable temporal structures. (2) Decoupling Mechanism : We employ time series decomposition to model seasonal and trend components separately. The stable characteristics of periodic and seasonal patterns enhance the model's robustness against temporal shifts.

Reference:

[1] Park, Hyeonjin, et al. "Metropolis-hastings data augmentation for graph neural networks." NeurIPS 2021.

[2] Wu, Ying-Xin, et al. "Discovering invariant rationales for graph neural networks." ICLR 2022.

[3] Zhao, Tong, et al. "Data augmentation for graph neural networks." AAAI 2021.

[4] Ma, Jianxin, et al. "Disentangled graph convolutional networks." ICML 2019.

[5] Li, Haoyang, et al. "Ood-gnn: Out-of-distribution generalized graph neural network." TKDE 2022.

Thank you very much for your appreciation and comments on the paper. Your comments are crucial to improving the quality of the manuscript.

W1 & Q2. Efficiency Analysis

We sincerely apologize for any misunderstanding caused. Below, we report the efficiency comparison between the proposed method and several advanced models on the CA dataset. The efficiency metrics include memory usage, training time per epoch, total training time, and the amount of memory used. We can find that our model achieves competitive performance while maintaining high efficiency

W2. OOD Setting

This issue is addressed in Appendix E.1. For models such as GWNet and AGCRN, they employ an adaptive graph learning enhancement technique, which generates meaningful node representations for each node but also causes the parameter scale to become coupled with the size of the graph structure. To address this, we have removed this technique.

Q1. Spatio-temporal prompt

This technique uses embeddings to encode the day-of-week and timestamp-of-day information. These prior knowledge capture the periodic patterns in spatiotemporal data, which are relatively stable and help improve the model's generalization ability against spatiotemporal shifts.

Q3. Traditional GCN

Whether GCN is necessary in the field of spatiotemporal prediction has long been a topic of debate. However, many studies in the spatiotemporal domain have consistently shown that GCN-based models outperform those that only consider temporal dependencies, as they introduce an inductive bias between nodes. In spatiotemporal OOD tasks, the dense message-passing mechanism of GCNs demonstrates sensitivity to spatiotemporal shifts. Therefore, we focus on enhancing the robustness of spatiotemporal interactions and propose a centralized interaction mechanism as an alternative solution.

Q4. Message Interference Mechanism

In lines 188 to 197, we describe the random sampling process for the stochastic masking matrix. Specifically, we first create $M$ learnable probability vectors and normalize it, with the result denoted as {$p_1^\prime, p_2^\prime,\dots, p_M^\prime$}, where ${p}_i^\prime \in \mathbb{R}^N$ with $i \in$ {$1, 2, \cdots, M$} means $i$-th probability vector. Then, for $i$-th probability vector, we can establish a binomial distribution, which is denoted as $\mathcal{M}\left({p}_i^\prime;s\right)$. Using this distribution, we can sample a masking indices $\widetilde{g}_i \sim \mathcal{M} \left({p}_i^\prime;s \right) \in$ {0,1}$^N$, where $s\in\left(0,N\right)$ indicates the number of sample hits (i.e. the number of values equal to 1 in $\widetilde{\boldsymbol{g}}_i$). Finally, we create $K$ replicas of $\widetilde{\boldsymbol{g}}_i$. As a result, we can obtain a mask matrix with log operation, and the output is denoted as $\mathbf{G}_i=\log\left(\left[\widetilde{\boldsymbol{g}}_i,\widetilde{\boldsymbol{g}}_i,\dots,\widetilde{\boldsymbol{g}}_i\right]\right)\in\lbrace-\infty,0\rbrace^{K\times N}$ and $\mathbf{G}_i$ is used to interfere with the message process. Similarly, we ultimately perform multiple rounds of perturbations on the message-passing mechanism using $M$ interference matrices. To avoid further confusion, we will refine this section to improve clarity and ensure a more precise presentation.