T-Graphormer: Using Transformers for Spatiotemporal Forecasting

Thank you for dedicating your time to carefully review our work and provide valuable feedback! We are glad you also believe in using this approach in other domains. Here, we address your comments in detail:

This paper claims that T-Graphormer performs well on spatial-temporal data; however, only traffic datasets were used in the experiments, so testing on additional datasets is encouraged. Additionally, the experiments focus primarily on main results and ablation studies, and further experiments, such as hyperparameter analysis, are recommended.

• Dataset Selection: We acknowledge the importance of evaluating T-Graphormer across a broader set of traffic datasets. We are actively working to include PEMS03, PEMS04, PEMS07, and PEMS08 in our experiments. The results will be updated in the revised manuscript once these evaluations are complete.
• Hyperparameter Study We have conducted extensive hyperparameter testing. For instance, in Tables 4 and 5 of the Appendix, we report the best hyperparameters of different T-Graphromer model variants on the two traffic datasets. In Table 6, we report the test metrics of these variants. We also find that larger variants of T-Graphormer are likely to overfit, and this behaviour is consistent with the recent findings on the empirical scaling laws for training LLMs [1].

This paper would benefit from incorporating more recent citations on state-of-the-art Transformer research, particularly in the baseline comparisons. Adding newer relevant works, such as DSformer [Yu, 2023] and LDT [Feng, 2024], is recommended.

We apologize for the oversight in not including these recent related works in our literature review and baseline comparisons. We have incorporated advanced baselines, including PDFormer and STAEformer, into our comparisons (see Figure 2 and Section 2.1). The results demonstrate that T-Graphormer consistently outperforms these models.

Even though detailed information about the implementation are provided, no code provided would still make it hard to evaluate the results.

As stated in our Reproducibility Statement, we include the ave included links to the anonymized code repository, the datasets, and the pre-trained model weights in Section A.1.

Q1. The encoding strategy in the methodology seems primarily focused on a single data dimension. However, spatial-temporal data inherently involves complex interactions between spatial and temporal dimensions. Given that the embedding preprocessing is focused solely on the temporal dimension, how could the current method effectively capture dependencies across both dimensions?

In fact, T-Graphormer has more structural encoding methods in the spatial dimension.

• By incorporating the centrality encoding into node features, and using the shortest path distance to adjust attention bias, the model can update representation by selectively paying attention to more important nodes.
• In the temporal dimension, we add learnable positional encoding to the initial node features. This has proven useful in the original Transformer architecture [2], and the ViT architecture [3, 4] where data structures are more complex.

Q2. The attention mechanism is indeed effective at capturing correlation ... This would help address the lack of sufficient Transformer or attention-based baselines for comparison in the experiments.

This has been done in a similar manner by STSGCN [5], where they also try to simultaneously capture spatial and temporal dependencies with GNNs. They do so by connecting the nodes across time steps. In comparison, T-Graphormer processes the input as a sequence. This distinction underscores a fundamental difference in how the two models integrate spatial and temporal information.

Could you provide a brief summary of how the graph was generated and the adjacency matrix constructed? While I understand the implementation follows another paper, it would be helpful if these details were mentioned directly in your paper for clarity.

To construct the sensor graph, we compute the pairwise road network distances between sensors and build the adjacency matrix using thresholded Gaussian kernel. We have added these details in our appendix.

We sincerely thank the reviewer for the thoughtful and detailed feedback, which has significantly improved our manuscript in terms of its clarity, rigour, and impact. We hope our detailed responses and manuscript revisions address all concerns. We look forward to further correspondence.

[1]Kaplan, J. (2020). Scaling laws for neural language models. arXiv preprint.

[2]Vaswani, A. (2017). Attention Is All You Need. NeurIPS.

[3]Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.

[4]Feichtenhofer, C. (2022). Masked autoencoders as spatiotemporal learners. NeurIPS.

[5]Song, C. (2020). Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. AAAI.

Thanks the authors for their efforts. I have carefully read all the replies and decided to maintain my score. I acknowledge your efforts to incorporate additional traffic datasets (PEMS03, PEMS04, etc.). Including these datasets will indeed enhance the generalizability and robustness of the proposed method. However, as these results are not yet included, the current evaluation remains limited to 2 traffic datasets, which constrains the paper's broader applicability.

Thank you for dedicating your time to carefully review our work and provide valuable feedback! We are glad you like Figure 2 and the ablation study. Here, we address your comments in detail:

The baseline methods are outdated. Authors need to compared this model with more advanced and latest baselines, instead of SVM and FC-LSTM. Additionally, more datasets should be included. For example, you might include STID, MegaCRN and PDFormer ... For datasets, more traffic datasets, such as PEMS03, PEMS04, PEMS07 and PEMS08 can be included. It is not sufficient to only use two datasets to demonstrate the effectiveness of your model.

• Baseline Comparisons: We apologize for the oversight in not including these recent related works in our literature review and baseline comparisons. We have incorporated advanced baselines, including PDFormer and STAEformer, into our comparisons (see Figure 2 and Section 2.1). The results demonstrate that T-Graphormer consistently outperforms these models, highlighting the robustness and effectiveness of our proposed method.
• Dataset Selection: We acknowledge the importance of evaluating T-Graphormer across a broader set of traffic datasets. We are actively working to include PEMS03, PEMS04, PEMS07, and PEMS08 in our experiments. The results will be updated in the revised manuscript once these evaluations are complete.

The writing should be improved. The authors use 'spatial temporal forecasting', 'multivariate time series forecasting', 'traffic prediction' in the introduction and section. Although these terms are correlated, they are different and they might be better to focus on one specific task while writing this paper. Since your data is traffic, it is better to frame it as a traffic prediction problem and emphasize it in your introduction.

We would appreciate clarification on which section the reviewer finds unclear.

We have carefully written our introduction such that the scope of discussion is gradually narrowed to traffic prediction, which is the specific task we are evaluating T-Graphormer on. We believe that since T-Graphormer is a Transformer architecture directly applied onto traffic data with minimal spacetime inductive biases, it can be readily applied to other data modalities. In our work, we are introducing this general framework. This has been made clearer in Section 6 of our manuscript.

The methodology itself is not innovative. For example, authors do not make modifications to the traditional transformer structure to make improvement. Many papers adjust the attention mechanism to better capture temporal/spatial correlations. Why you only use the original transformer?

We respectfully disagree with the assertion that the methodology lacks innovation. We not only borrow the structural encoding methods from Graphormer and extend them along the temporal dimension but also examine the effects of added special tokens (namely $\textt{cls} and \textt{graph}) and prediction layers (fully connected and dilated causal). Specifically, the attention mechanism is adjusted in our spatial encoding method (defined in Equation 7). We decide to use the original Transformer with added structural encoding methods to fully utilize its global attention mechanism such that the spatial and temporal inter-correlations are modelled simultaneously.

Why you choose STEP as the baseline? It seems that your model does not have a higher performance.

We selected STEP as a baseline because it has been shown to outperform PDFormer and STAEformer on the PEMS-BAY and METR-LA datasets. Our results demonstrate that T-Graphormer consistently outperforms STEP in predicting the next 12 time steps (1-hour window), which we believe underscores the model's efficacy for longer-range traffic predictions. We discuss the performance gap in short-range predictions in Section 5.3 of our manuscript.

Since your title is spatial temporal forecasting, do you agree that more spatial temporal dataset should be concluded, such as weather dataset and crime dataset?

We agree that T-Graphormer should be evaluated on other spatiotemporal datasets, such as weather and crime datasets. While our current scope focuses on introducing this framework for traffic prediction, future work will explore its applicability to diverse spatiotemporal data modalities.

We sincerely thank the reviewer for the thoughtful feedback, which has significantly improved our manuscript in terms of its clarity, rigour, and impact. We hope our detailed responses and manuscript revisions address all concerns. We look forward to further correspondence.

Thank you for dedicating your time to carefully review our work and provide valuable feedback! Here, we address your comments in detail:

W1.Key points lack details. While the existing content and formulas in Graphormer are well-articulated, The description of the novel contributions is not sufficiently detailed. In Section 4.2, the paper only presents the modified transformer method for T-Graphormer only in brief with the expression l=T'*N. In Section 4.3, the paper doesn't clearly convey the formulas for special tokens. Could you provide detailed mathematical formulas and subsequent explanations for these original contributions?

• In Section 4.2, we not only present the flattening process of the graph sequence ($l = T' \times N$), but also explicitly detail the structural and positional encoding extensions. We accompany these descriptions with Figure 2, visualizing the model architecture with carefully selected colours matching that of nodes in Figure 1. We also highlight the novelty of these encodings compared to the original Graphormer formulation.
• We apologize for not elaborating on the details of the special tokens and causing confusion. To clarify (we have added these details in our revised manuscript):
  • The $`cls`$ token is implemented similarly to the virtual node in Graphormer. Its purpose is to aggregate information across the entire sequence and propagate it back to individual tokens.
  • The $`graph`$ token functions analogously to the $`sep`$ token in BERT (Devlin et al., 2018) [2], serving as a delimiter for graph signals at different time steps.

W2. Lack of recent related work. The related work mentioned in Chapter 2, Related Works, basically references studies prior to 2020. There's also a lack of references to newer traffic prediction models. Including recent works such as STAEformer [1] would strengthen the baselines.

We apologize for the oversight in not including these recent related works in our literature review and baseline comparisons. We have added PDFormer and STAEformer in Figure 2 and Section 2. 1. These additions position our work within the broader context of modern traffic prediction models and highlight the improvements our method achieves over these benchmarks.

W3. Lack of visualization. While the main claim of the paper is that directly modeling spatiotemporal correlations is better than modeling them separately, there is no accompanying visualization or case study to substantiate this claim, reducing the argument's credibility. Could you provide a visualization of an attention example for a particular sensor to illustrate that the T-Graphormer indeed learns spatiotemporal patterns effectively compared to previous methods?

We definitely agree that visualizing the model's prediction can provide further credibility and a more intuitive understanding of its performance. Since we flatten the entire graph sequence, we find it extremely difficult to interpret the attention scores on 3900 ($325 \times 12$) tokens in PEMS-BAY and 2484 ($207 \times 12$) tokens in METR-LA. We instead provide a more interpretable visualization in Figures 3 and 4. These figures demonstrate T-Graphormer's spatiotemporal modelling capability by showcasing predictions for individual sensors across 587 time steps. This alternative approach offers a practical perspective on the model's performance.

Minor remarks: In the last paragraph of 4.2, the symbol upright phi appears without prior definition.

Thank you for spotting our mistake, this has been corrected in our manuscript.

We sincerely thank the reviewer for the thoughtful feedback, which has significantly improved our manuscript in terms of its clarity, rigour, and impact. We hope our detailed responses and manuscript revisions address all concerns. We look forward to further correspondence.

Thank you for dedicating your time to review our work and provide valuable feedback. We address your comments in detail:

Strengths: Two novel spatial feature extraction methods are proposed in this paper, one for encoding the in-degree and out-degree of the graph structure and the other for encoding the shortest travel distance between graph nodes.

While these structural encoding techniques are crucial to T-Graphormer's modelling capabilities when the graph sequence is flattened (see Section 5.4, Ablation Studies), we did not introduce these methods. They were first proposed in Graphormer by Ying et al. (2021) [1]. As stated in Section 4, our contribution extends these structural encoding methods to the temporal dimension, building on their foundation.

There have been several methods [1][2] that apply transformer architecture to spatio-temporal forecasting. The idea of using transformers to simultaneously capture temporal and spatial correlations is not sufficiently novel.

We respectfully disagree. While prior methods leverage the Transformer architecture in their encoder models, T-Graphormer is fundamentally different in how it unifies spatiotemporal learning.

As outlined in Section 2.1 (Related Work), existing methods use separate modules for temporal and spatial learning. Regarding the references the reviewer provided,

• PDFormer splits attention heads into Semantic Spatial Attention, Geographic Spatial Attention, and Temporal Self-Attention.
• STAEformer uses Temporal transformer layer followed by a Spatial transformer layer (Equation 7).

Our work directly applies the Transformer architecture to spatiotemporal data, acknowledging that dependencies exist across dimensions, not just within them. By treating spatiotemporal data holistically, T-Graphormer effectively models these interdependent correlations without separating temporal and spatial dimensions.

Section 4.3 does not adequately explain the setting of additional tokens. These tokens are typically used in the NLP domain, but the authors fail to clarify their significance in spatio-temporal forecasting or provide details on their implementation.

We acknowledge the need for clearer explanations of the additional tokens.

• The $`cls`$ token is implemented similarly to the virtual node in Graphormer. Its purpose is to aggregate information across the entire sequence and propagate it back to individual tokens.
• The $`graph`$ token functions analogously to the $`sep`$ token in BERT (Devlin et al., 2018) [2], serving as a delimiter for graph signals at different time steps.

The experimental design has shortcomings. First, the dataset selection is insufficient ... Finally, the hyperparameter experiments do not sufficiently explore hyperparameter values.

1. Dataset Selection: We are actively expanding our experiments to include additional traffic speed datasets and will update our results accordingly.
2. Baseline Comparisons: Advanced Transformer-based baselines have been added to Table 2, and T-Graphormer maintains superior performance on the PEMS-BAY dataset and METR-LA dataset. We apologize for the oversight in not including these recent related works in our literature review and baseline comparisons.
3. Hyperparameter Exploration: We are unsure which specific hyperparameter values the reviewer is referring to. Additional clarification would help us refine this aspect.

The manuscript's structure is also not well-balanced. Chapter 4, which describes the main method, occupies only one and a half pages, while other sections take up considerably more space.

We believe there is no universally "correct" manuscript structure. Our framework is designed to be simple and intuitive, warranting a concise description in Chapter 4. The remaining sections focus on contextualizing, validating, and analyzing the method. However, if there are specific aspects of Chapter 4 that the reviewer finds unclear, we are happy to provide further elaboration.

The language presentation is rough. The adjacency matrix dimension in Section 3.1 is incorrect. Section 4.1 consists of only two sentences and lacks a period. Additionally, the summary in Section 6 is confusing and poorly presented.

We sincerely apologize for these errors and have corrected them in the updated manuscript. Regarding Section 6, we would greatly appreciate specific feedback on which parts are confusing to address them more effectively. Nonetheless, we have revised and improved the clarity of this section.

Once again, we appreciate your feedback and look forward to further correspondence.

[1]. Do transformers really perform badly for graph representation?. (Ying et al., NeurIPS 2021)

[2]. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., NAACL 2019)

Dear Reviewer SDZa,

As the discussion period comes to an end, we would love to have additional feedback on our rebuttal response so we can continue improving the manuscript.

Have our comments addressed your concerns about our work? Do you have further questions?

Thank you again for your time.

Best, Authors

Dear Reviewer SDZa,

Thank you for your response. We sincerely appreciate your feedback and the opportunity to further clarify our contributions.

• Hyperparameter Experiment: We apologize if our description led to any misunderstanding. As detailed in Tables 4 and 5, for each dataset and model variant (mini, small, medium, causal mini, causal small, causal medium), we conducted an extensive hyperparameter search. Specifically, we evaluated the following parameters:

  learning rate, gradient clipping, weight decay, dropout, and layerwise decay

  Among these, we observed that learning rate, layerwise decay, and weight decay (in that order) had the most significant impact on model performance during training.

  If you feel that further details are necessary, we would be happy to expand on this as we stored all of our runs on Weights and Biases.

• Contribution: To the best of our knowledge, this work is the first to directly apply a Transformer architecture to spatiotemporal data with minimal spacetime inductive biases.

  Our proposed method, T-Graphormer, not only improves RMSE by 10% on the PEMS-BAY dataset and MAPE by 10% on METR-LA compared to existing approaches but also reframes spatiotemporal forecasting. By modelling spatiotemporal data holistically, T-Graphormer identifies interdependent correlations without explicitly separating temporal and spatial dimensions. Future research can easily extend T-Graphormer by incorporating domain knowledge into the architecture, such as using adaptive spatiotemporal embeddings from STAEformer.

We thank you again for your feedback, and we hope this response addresses your concerns. Thank you again for your time.

Best,

Authors