Language Model Empowered Spatio-Temporal Forecasting via Physics-Aware Reprogramming

We sincerely thank the reviewer for the valuable feedback. We group related points in Weaknesses (W) and Questions (Q) as follows.

W1: The proposed decomposer is not clearly described. Since the decomposition is a common tool in time series analysis, there lacks a discussion of why the decomposed components are physics aware.

[Response] Thank you for your valuable feedback. Our decomposer fundamentally differs from traditional decomposition methods. Below we provide a more detailed explanation of our decomposer and its physics-aware characteristics.

Our evolutionary decomposition method builds upon the principles of Dynamic Mode Decomposition and Koopman operator theory, both of which are rooted in capturing the intrinsic dynamics of a system through a linear approximation of its non-linear behavior. By analyzing the dominant modes, our evolutionary decomposition can uncover patterns that are both interpretable and consistent with the underlying physical mechanisms, providing valuable insights for PLM that can improve predictive accuracy.

As many studies have demonstrated, our DMD-based evolutionary decomposition extracts both spatial coherence and temporal dynamics, which represent key phenomena such as propagating waves, oscillatory behaviors, and decay patterns that are closely linked to dominant physical processes [1, 2]. The spatial modes obtained through decomposition reveal coherent structures inherent to the physical system. Importantly, the spatial modes retain the relationships dictated by the governing physical laws, ensuring the decomposition aligns with the true behavior of the system. For instance, it has been shown that DMD can uncover vortex shedding and periodic oscillations in fluid dynamics [2]. These patterns align closely with the Navier-Stokes equations, capturing the fundamental physical dynamics governing fluid motion [3]. Furthermore, each mode identified by evolutionary decomposition is associated with a temporal eigenvalue, which encodes the growth, decay, or oscillatory behavior of that mode. These eigenvalues reflect critical physical phenomena such as natural frequencies, damping rates, and stability margins.

Unlike PCA or Fourier decomposition, which primarily extract statistical patterns, our approach ensures the temporal evolution and spatial structure are inherently connected, providing a physically meaningful framework for analyzing and predicting system behavior. Our method ensures that each mode is tied to the system’s dynamics, making it particularly well-suited for spatio-temporal datasets governed by physical laws.

We will add these discussions in the revised manuscript. Thank you again for your thoughtful feedback!

[1] Rowley C W, Mezić I, Bagheri S, et al. Spectral analysis of nonlinear flows[J]. Journal of fluid mechanics, 2009, 641: 115-127.

[2] Tu J H. Dynamic mode decomposition: Theory and applications[D]. Princeton University, 2013.

[3] Kutz J N, Brunton S L, Brunton B W, et al. Dynamic mode decomposition: data-driven modeling of complex systems[M]. Society for Industrial and Applied Mathematics, 2016.

W2: The usage of the reconstruction matrix is ambiguous. Further ablation studies are needed.

[Response] Thank you for your valuable feedback, and we are sorry for any lack of clarity regarding the usage of the reconstruction matrix.

As described in lines [198–208], reconstruction plays a pivotal role in DMD-related techniques [1,2], ensuring that the extracted modes faithfully capture the system's underlying dynamics while eliminating redundant noise. In our approach, we enhance this process by prioritizing the most dominant modes, enabling a more precise representation of key spatio-temporal patterns.

We agree that additional ablation studies are crucial to further clarify the impact of the reconstruction matrix on model performance. To address this, we conducted experiments to systematically evaluate its role and assess its influence on the overall results, following the setting of the ablation study in the paper.

w/o reconstruction: RePST with only dynamic components (X_dyn) as X_dec

w/o dynamics: RePST with only reconstruction data (X_rec) as X_dec

As can be seen in the table above, the model performance declines obviously when we remove either dynamic components (X_dyn) or reconstruction data(X_rec). The ablation studies confirm that the reconstruction matrix is essential for maintaining high performance, alongside the dynamic’s component. The results also demonstrate the necessity of integrating both components to fully realize the advantages of the RePST framework.

Thank you once again for bringing this to our attention!

Dear Reviewer EE3c,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best,

Authors

We sincerely thank the reviewer for the valuable feedback. We group related points in Weaknesses (W) and Questions (Q) as follows.

W1 & Q1: I wonder why they didn’t opt for eigenvectors or simpler methods like PCA, which could also provide interpretable components. This makes me question whether the use of DMD here truly justifies the physics-informed label. Please clarify your reasoning behind using DMD over other methods like PCA or eigenvectors. Additionally, please provide more evidence or examples of how your decomposition method incorporates physical principles beyond pattern extraction. Is the physics-aware component primarily derived from DMD modes? If so, how does this differ from other decomposition methods like PCA or eigenvectors, which can also capture patterns in data?

[Response] Thank you for your detailed feedback and raising these important questions! Below we provide a point-to-point response for your comments.

Physics-Aware Justification

Although DMD is designed to extract dynamic modes directly from observational data, numerous studies have demonstrated its capability to identify propagating waves, oscillatory behaviors, and decay patterns, which are strongly associated with dominant physical phenomena [1,2,3]. For example, research has shown that DMD can reveal vortex shedding patterns and periodic oscillations in fluid dynamics [1]. These patterns are closely aligned with the Navier-Stokes equations, reflecting the intrinsic physical dynamics of fluid motion [2,3]. For this reason, DMD is particularly suitable for capturing physics-aware patterns from spatio-temporal data, such as traffic flow and air pollution, which exhibit similar regularities with those governing fluid dynamics. For instance, STDEN [4] and AirPhyNet [5] model the dynamics of traffic flow and air pollution as a continuous diffusion process via differential equations inspired by fluid dynamics modeling.

Why DMD Over PCA or Eigenvectors?

Unlike DMD, PCA or eigenvector-based methods are purely statistical tools that do not inherently account for the underlying physical behavior of a system. By analyzing the dominant modes, DMD can uncover patterns that are both interpretable and consistent with the underlying physical mechanisms, providing valuable insights for PLM that can improve predictive accuracy.

To validate our choice, we first conducted experiments by replacing the physical decomposer with PCA. This results in a noticeable decline in model performance, suggesting that PCA’s inability to benefit PLM in our context. Furthermore, when we randomly initialized the pretrained weights of the PLM, the impact on performance was minor, further indicating that PCA could not effectively leverage the pretrained knowledge. These findings underscore the superiority of DMD in generating physically consistent interpretations and unlocking pretrained model knowledge.

PCA_RePST: RePST with PCA as decomposer

PCA_random_RePST: RePST with PCA as decomposer and randomly initialize the weights of the PLM

W2 &Q4: The terminology of “reprogramming” feels overstated to me. How exactly does the authors’ approach to “reprogramming” the PLM differ from simply changing input structures? Were there any modifications to the PLM architecture or retraining steps involved?

[Response] Sorry for the confusion. The term “reprogramming,” as employed here and in prior studies [6] [7], does not refer to architectural changes or substantial retraining of the PLM. Instead, it describes the process of adapting or reformatting input data to align with the model’s existing architecture and capabilities. Rather than modifying the model’s internal structure or layers, reprogramming capitalizes on the inherent representational power of pretrained models by transforming input data to fit their original design. We hope this clarification addresses any misunderstanding regarding the use of the term “reprogramming.” Our approach aims to maximize the utility of pretrained models by adapting inputs and outputs, rather than altering the underlying architecture, making it an efficient and powerful method for cross-modal adaptation. Thank you again for your valuable feedback, which has helped us enhance the clarity and depth of our manuscript.

[6] Jin M, Wang S, Ma L, et al. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models[C]//The Twelfth International Conference on Learning Representations.

[7] Yang C H H, Tsai Y Y, Chen P Y. Voice2series: Reprogramming acoustic models for time series classification[C]//International conference on machine learning. PMLR, 2021: 11808-11819.

W3 & Q5: I wonder if there is an inconsistency in how the baseline models were evaluated. Please clarify whether the baseline models were evaluated using the same augmented data as REPST, and if not, provide results of baselines run on the augmented data for a fair comparison. Have the baseline models been tested with the same augmented data as REPST? If not, how would the results compare under such conditions?

[Response] Thanks for your constructive suggestion! We acknowledge the importance of evaluating baseline models under consistent conditions. The time series forecasting models (e.g., PatchTST, iTransformer) were originally designed to process multivariate time series data structured as (B, L, C), where B is the batch size, L is the sequence length, and C is the number of variables. Adapting these models to spatio-temporal forecasting tasks—where the data is structured as (B, L, N, C'), with N denotes the number of spatial nodes and C' is the decomposed feature dimension—requires treating N as variable dimension (i.e., the dimension C in time series data (B, L, C)). This adaptation overlooks decomposed features (C'), preventing these models from fully leveraging the augmented data. To evaluate the impact of augmented data on these time series forecasting models, we explored an alternative adaptation by reshaping the spatio-temporal data into (B*N, L, C'), allowing the models to access the augmented features. The results showed that due to the lack of explicit spatial modeling capabilities, these models performed even poorly when provided with augmented data.

For state-of-the-art spatio-temporal forecasting models, we conducted additional experiments using the same augmented data. While some improvements were observed, the gains were limited. These models struggled to effectively utilize the augmented data, likely because they fail to interpret and harness the physics-aware knowledge encapsulated in the augmented features.

The above results confirm that the physics-aware augmented data provides unique benefits when integrated with PLMs in our RePST framework. In contrast, other models that solely rely on fine-grained features fail to effectively utilize this information. We hope this explanation resolves your concerns and underscores the strengths of our method.

Thank you once again for your valuable feedback, which is instrumental in enhancing our analysis!

Dear Reviewer Xa2y,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best,

Authors

Thanks for your prompt reply, which helps us further understand your concerns. We are pleased to hear that some of your questions have been addressed. Please find a point-to-point response to your new comments below.

[Q1]

However, I think you are mixing the concepts of physical laws and spatial-temporal dependencies. In another word, you are mixing the concepts of physics-informed machine learning and representation learning. Those fluid motions and examples you mentioned, do have underlying physical equations to explain the system dynamics. Those physics-aware or physics-informed methods generally use data-driven methods to approach some parameters of the physic formula (usually some hard-to-solve PDE equations). In your case, those decomposers are just some representations learning to tell the spatial and temporal information, which is the abuse of terminology physics-aware or physics-informed field.

[Response]

We first highlight the distinction between physics-aware and physics-informed approaches [1,2,3].

• Physics-informed methods explicitly integrate physical equations (e.g., PDEs) into the modeling process, often to learn specific parameters or solve otherwise intractable equations. Examples include Physics-Informed Neural Networks (PINNs) that embed governing equations directly into the loss function.

• Physics-aware methods, on the other hand, do not explicitly incorporate physical equations but are designed to recognize, extract, or align with the underlying physical behaviors or dynamics from data. They use domain knowledge to ensure the results are physically meaningful without directly solving the equations. For example, PhAST [1] learns physics-aware embeddings without explicitly incorporating physical equations. But instead, it relies on features that align closely with physical principles. This approach is similar to DMD, which approximates the dynamics of complex systems in a data-driven manner without explicitly solving physical equations.

So DMD is not physics-informed in the strict sense because it does not incorporate governing equations or constraints directly into its formulation. Instead, its physics-aware nature lies in its ability to discover and align with physical patterns from data, making it highly valuable in contexts where explicit equations are unknown or impractical to solve.

Moreover, we respectfully invite the reviewers take a look at the Section 3 and 3.4 of a recent survey paper [4], where Koopman Theory methods (e.g., DMD) are also regarded as physical knowledge, although such methods do not explicitly incorporate physical equations. Referring to DMD as 'physics-aware' is therefore reasonable and consistent with the distinction between data-driven methods that identify physically consistent patterns and those that directly embed physical laws.

We appreciate your thoughtful comments and agree that revising our terminology can help avoid misunderstandings. Based on your feedback, we propose adding more details in related work to further discuss the difference between our method and physics-informed learning. Thank you for helping us refine our explanation.

[1] Duval A, Miret S, Bengio Y, et al. PhAST: Physics-Aware, Scalable, and Task-Specific GNNs for Accelerated Catalyst Design[J]. Journal of Machine Learning Research, 2024.

[2] Gurbuz S Z. Physics-Aware Machine Learning for Dynamic, Data-Driven Radar Target Recognition[C]//International Conference on Dynamic Data Driven Applications Systems. Cham: Springer Nature Switzerland, 2022: 114-122.

[3] Zheng Y, Zhan J, He S, et al. Curricular contrastive regularization for physics-aware single image dehazing[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 5785-5794.

[4] Meng C, Seo S, Cao D, et al. When physics meets machine learning: A survey of physics-informed machine learning[J]. arXiv preprint arXiv:2203.16797, 2022.

[Q2]

How about the performance for those time series data if they have C and C' features, or actually RePST just uses C' features?

[Response] Thanks for your insightful feedback! We are willing to conduct experiments on RePST when just uses C'. Similarly, we reshape the input data as (B*N, L, C',1) like other time series models. As can be seen in the table below, the model performance decreases. It demonstrated that only leveraging C' is not effective for RePST. However, compared to other time series models like PatchTST and iTransformer, RePST still achieves better performance even when spatial information is not modeled (using only C' ).

Thanks again so much for your insightful feedback and helpful suggestions regarding our work!

Q5: The inconsistent of E’ in Figure 2 and text

[Response] Thanks for your valuable feedback and we apologize for the confusion. We will fix the inconsistent expression in the revised manuscript.

Q6: What model does "HI" represent in line 285?

[Response] Thank you for your comments, and we apologize for the confusion caused. "HI" refers to "Historical Inertia" (HI) [9], a baseline model designed to leverage the natural continuity of historical data without relying on trainable parameters. HI directly uses the historical data point closest to the prediction target within the input time series as the forecasted value. HI capitalizes on the inherent persistence of historical patterns, making it a simple yet effective benchmark for time series forecasting tasks. While often overlooked, this method serves as a valuable reference point for assessing the performance of more complex models.

[9] Cui, Yue, Jiandong Xie, and Kai Zheng. "Historical inertia: A neglected but powerful baseline for long sequence time-series forecasting." Proceedings of the 30th ACM international conference on information & knowledge management. 2021.

Q7: Since the Koopman theory-based evolutionary matrix can describe the evolution of the system over time t, is it possible to use matrix A directly for prediction? If so, it is recommended to include it in the comparative experiments.

[Response] Thanks for your useful suggestion! Koopman theory-based evolution matrix is not commonly used for prediction in existing research. Instead, it is primarily employed as a tool for dynamic analysis rather than as a primary means of forecasting. Nonetheless, we are willing to conduct experiments that directly use the matrix A for prediction. As shown, such simple prediction methods fall significantly short of the performance achieved by our model RePST.

We sincerely thank you for your valuable feedback and constructive suggestions on our work. If possible, we kindly ask you to reconsider the ratings. Should there be any further questions or concerns, we would be more than happy to address them in detail.

We sincerely thank the reviewer for the valuable feedback. We group related points in Weaknesses (W) and Questions (Q) as follows.

W1: The statement that "the rich physical semantic information can boost the pretrained physical knowledge of PLMs" (Line 288-231) lacks experimental or theoretical evidence demonstrating that it was specifically the physical knowledge of PLMs that contributed to the results. It seems more likely that the physical methods applied simply extracted features that were more reasonable and of finer granularity.

[Response] Thanks for your insightful comments. We conducted experiments in our paper (Figure 5) to test the effectiveness of PLMs by replacing PLMs with trainable attention layers. To further verify the effectiveness of pre-trained knowledge in PLMs, we conducted additional experiments as follows: First, we examined whether the decomposed features could benefit baseline models. Specifically, we fed the decomposed feature into two state-of-the-art spatio-temporal forecasting models, i.e., GWNet and STID. The results showed only limited performance improvements, suggesting that even advanced spatio-temporal forecasting models cannot fully exploit these decomposed physical features. This finding indicates that the observed performance boost in our model cannot be solely attributed to fine-grained feature engineering. Second, we removed the pre-trained knowledge in the PLM by randomly initializing its weights. As can be seen in the following table, this led to a significant performance decline, highlighting the importance of pretrained knowledge in comprehending and utilizing the decomposed physical semantics.

GWNet_decomposer: GWNet using decomposed features

STID_decomposer: STID with decomposed features

RePST_random: randomly initialize the weights of the PLM in our model

In conclusion, these experiments collectively demonstrate that the pretrained knowledge embedded in PLMs is a critical factor driving the performance gains, rather than solely the extraction of finer-grained features through physical decomposition. We hope this clarifies our approach and addresses your concerns.

W2: The paper, i.e., abstract part, mentions that the physics-aware decomposer allows PLM to understand complex spatiotemporal dynamics using a divide-and-conquer strategy. How exactly does this relate to the divide-and-conquer approach?

[Response] Sorry for the confusion. The “divide-and-conquer” approach mentioned in the abstract specifically refers to the role of the physics-aware decomposer in simplifying the modeling of complex spatio-temporal dynamics. As explained in the paper (lines 65–66), spatio-temporal dynamical systems are significantly more complex and densely coupled than standard time series data, making it challenging to model them directly. To address this, the decomposer divides input signal into multiple sub-components, each characterized by distinct and interpretable spatio-temporal semantics. By isolating these components, the PLM can focus on comprehending and modeling specific patterns more effectively, rather than contending with the overwhelming complexity of the system. This process embodies the essence of divide-and-conquer strategy, where a complex problem is broken down into manageable sub-problems, each of which can be addressed independently. In this way, the decomposer enables the PLM to concentrate on distinct aspects of the spatio-temporal dynamics, ultimately enhancing overall comprehension and performance.

W3&Q1: The experimental setup is limited to a short prediction length, with no results for other prediction lengths. Does REPST perform well only for short prediction lengths?

[Response] Thank you for your valuable and constructive feedback. As is common in spatio-temporal forecasting research (e.g.,[1], [2], [3], [4], [5].), our experiments focused on relatively short prediction lengths to align with standard evaluation protocols in the field. However, we acknowledge the importance of exploring longer prediction horizons to provide a more comprehensive assessment of model performance. In response, we conducted additional experiments with a prediction length of 96 across four datasets (OOM: Out-Of-Memory).

As can be seen, RePST consistently outperformed powerful baseline models even in long-term predictions, underscoring the robustness of our approach across varying prediction lengths. We understand the need for broader experimental coverage and are committed to extending our evaluations to include all baselines and datasets for longer prediction lengths in the revised manuscript. We hope this addresses your concern, and we sincerely thank you again for your valuable suggestion.

[1] Yuan Y, Shao C, Ding J, et al. Spatio-temporal few-shot learning via diffusive neural network generation[C]//The Twelfth International Conference on Learning Representations. 2024.

[2] Cini A, Marisca I, Zambon D, et al. Taming local effects in graph-based spatiotemporal forecasting[J]. Advances in Neural Information Processing Systems, 2024, 36.

[3] Xia Y, Liang Y, Wen H, et al. Deciphering spatio-temporal graph forecasting: A causal lens and treatment[J]. Advances in Neural Information Processing Systems, 2024, 36.

[4] Li Z, Xia L, Xu Y, et al. GPT-ST: generative pre-training of spatio-temporal graph neural networks[J]. Advances in Neural Information Processing Systems, 2024, 36.

[5] Li Y, Yu R, Shahabi C, et al. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting[C]//International Conference on Learning Representations. 2018.

[6] Wu Z, Pan S, Long G, et al. Graph wavenet for deep spatial-temporal graph modeling[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019: 1907-1913.

W4: The experimental results do not include standard deviations and there is no mention of random seeds used, making it difficult to assess the statistical significance and reproducibility of the results.

[Response] Thank you for your thoughtful suggestion! We recognize the importance of statistical significance and reproducibility in evaluating experimental results. To clarify, all reported performance metrics for RePST are averaged over three independent runs with different random seeds to ensure reliability in the results. While standard deviations are not currently included in the paper, we will incorporate these details in the future revision to enhance the statistical rigor of our work. Furthermore, we have prepared the codebase for our experiments, to facilitate full reproducibility of our results. This code will be publicly released soon to ensure transparency and support further exploration by the community. Thank you again!

Dear Reviewer i6Kn,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best,

Authors

[Q3-1] Over-claiming and concept clarity: The paper tends to over-claim its contributions. Important concepts such as reprogramming and divide-and-conquer remain ambiguous despite being crucial to the paper’s framework. This lack of clarity contributes to the paper’s complexity and readability issues.

[Response] The concepts of “reprogramming” [6,7] and “divide-and-conquer” [8] are widely recognized in the field. Our use of these terms aligns with their conventional meanings, and we do not claim to redefine or expand them beyond their established scope. Therefore, we respectfully disagree with the assertion that our work over-claims the contributions or makes any ambiguous statements on these two concepts. These concepts were build upon existing studies, and we ensured their usage was consistent with prior literature.

[6] Jin M, Wang S, Ma L, et al. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models[C]//The Twelfth International Conference on Learning Representations.

[7] Yang C H H, Tsai Y Y, Chen P Y. Voice2series: Reprogramming acoustic models for time series classification[C]//International conference on machine learning. PMLR, 2021: 11808-11819.

[8] Deng J, Chen X, Jiang R, et al. Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting[J]. IEEE Transactions on Knowledge and Data Engineering, 2024.

[Q3-2] The notion of “interpretability” is frequently mentioned, yet it is not convincingly demonstrated how this is achieved within the context of the work.

[Q4-1] Intuition of the physics-aware component: As mentioned in my first weakness and also noted by Reviewer Xa2y, the intuition and motivation behind the incorporation of the physically aware component are not sufficiently explained.

[Response]

Physics-aware methods do not explicitly incorporate physical equations but are designed to recognize, extract, or align with the underlying physical behaviors or dynamics from data [1,2,3]. They use domain knowledge to ensure the results are physically meaningful without directly solving the equations. For example, PhAST [1] learns physics-aware embeddings without explicitly incorporating physical equations. But instead, it relies on features that align closely with physical principles. This approach is similar to DMD, which approximates the dynamics of complex systems in a data-driven manner without explicitly solving physical equations.

So DMD is not physics-informed in the strict sense because it does not incorporate governing equations or constraints directly into its formulation. Instead, its physics-aware nature lies in its ability to discover and align with physical patterns from data, making it highly valuable in contexts where explicit equations are unknown or impractical to solve.

Moreover, methods like DMD are also regarded as physical knowledge in recent studies [4], although such methods do not explicitly incorporate physical equations. Calling DMD physics-aware is therefore reasonable and aligns with the distinction between data-driven methods that respect physical principles and those that directly embed physical laws.

[1] Duval A, Miret S, Bengio Y, et al. PhAST: Physics-Aware, Scalable, and Task-Specific GNNs for Accelerated Catalyst Design[J]. Journal of Machine Learning Research, 2024.

[2] Gurbuz S Z. Physics-Aware Machine Learning for Dynamic, Data-Driven Radar Target Recognition[C]//International Conference on Dynamic Data Driven Applications Systems. Cham: Springer Nature Switzerland, 2022: 114-122.

[3] Zheng Y, Zhan J, He S, et al. Curricular contrastive regularization for physics-aware single image dehazing[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 5785-5794.

[4] Meng C, Seo S, Cao D, et al. When physics meets machine learning: A survey of physics-informed machine learning[J]. arXiv preprint arXiv:2203.16797, 2022.

[Q4-1] The author’s response to my question W1, which involved conducting experiments by ablating this component, did not convincingly demonstrate its significance or promise.

[Response] We conducted detailed experiments by removing the pre-trained knowledge in the PLM through random weight initialization, resulting in a performance degradation of up to 29.49%. Additionally, as reported in the ablation study section of our manuscript, replacing our decomposer with a transformer encoder led to a performance drop of up to 24.87%. These experiments underscore the critical role and significance of our decomposer.

We sincerely thank the reviewer for the valuable feedback. We group related points in Weaknesses (W) and Questions (Q) as follows.

W1: Some parts of the paper use vague descriptions. For instance, from Line 226-231, it could be better to be more specific in what the "reprogramming" does, and how "rich physical semantic information can boost the pretrained physical knowledge of PLMs".

[Response] Thanks for your time and valuable feedback! We apologize for the confusion.

The term “reprogramming,” refers to the process of adapting or reformatting input data to align with the model’s existing architecture and capabilities [1,2].

Our DMD-based decomposer inherently account for the underlying physical behavior of a system. For example, research has shown that DMD can reveal vortex shedding patterns and periodic oscillations in fluid dynamics [3]. These patterns are closely aligned with the Navier-Stokes equations, reflecting the intrinsic physical dynamics of fluid motion [4]. By analyzing the dominant modes, DMD can uncover patterns that are both interpretable and consistent with the underlying physical mechanisms, providing valuable insights for PLM that can improve predictive accuracy.

Leveraging such decomposition, we can disentangle the complex spatio-temporal system into more interpretable components, which can be described in human-like manners. This clear semantic information is inherently easy for PLMs to comprehend, which can boost the related pretrained physical knowledge of PLMs. Additionally, as can be seen in Figure 5 of our paper (table below), our decomposer which extracts rich physical semantic information contributes a lot when compared to directly modeling the spatio-temporal correlations by transformer encoder.

Thank you once again for your valuable feedback!

[1] Jin M, Wang S, Ma L, et al. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models[C]//The Twelfth International Conference on Learning Representations.

[2] Yang C H H, Tsai Y Y, Chen P Y. Voice2series: Reprogramming acoustic models for time series classification[C]//International conference on machine learning. PMLR, 2021: 11808-11819.

[3] Rowley C W, Mezić I, Bagheri S, et al. Spectral analysis of nonlinear flows[J]. Journal of fluid mechanics, 2009, 641: 115-127.

[4] Yu Y, Liu D, Wang B, et al. Analysis of Global and Key PM2. 5 Dynamic Mode Decomposition Based on the Koopman Method[J]. Atmosphere, 2024, 15(9): 1091.

W2: At lines 93-94, 467-468, and 537, the paper mentions that the spatio-temporal vocabulary enables PLMs to focus on relationships among 3D geometric space, but it is not further justified or explained in the main text. It is unclear why.

[Response] Thanks for your time and valuable feedback! We apologize for the confusion.

As can be seen in the ablation study part (Figure 5), we conducted experiments to test the effectiveness of the spatio-temporal vocabulary by replacing our selective spatio-temporal vocabulary and utilizes the dense mapping function from [1]. In the table from Figure 5, represented below, the model performance decreases obviously, which demonstrates that this fused vocabulary failed to provide insights for PLMs to comprehend 3D relationships (2D space and 1D time axis). In contrast, our selective spatio-temporal vocabulary is equipped with spatial-related and temporal-related words, which can well describe the spatio-temporal correlations and make the spatio-temporal semantic information clearer for PLMs to comprehend, leading to more precise modeling and prediction.

Thank you again for your insightful suggestion!

Q2: In terms of the experimental setup for traffic prediction, why is the prediction horizon 24, and not {3, 6, 12} like the baseline methods? This would allow for a more rigorous comparison in prediction performance. Additionally, for Figure 4, multiple prediction horizons are tested for the other datasets, but no data is provided on METR-LA and PEMS-BAY. Similarly, for Figure 5, only 3 datasets are shown, and the additional results are not found in the Appendix.

[Response] Thank you for your valuable feedback and for highlighting the need for more clarity.

The prediction horizons {3, 6, 12} are only standard setting for METR-LA and PEMS-BAY. To show performance on a longer horizon, we set the prediction horizon as 24. However, we are still willing to provide the few-shot experiment results with the prediction horizon 12 on METR-LA and PEMS-BAY, as follows:

For Figure 4, we conducted experiments on Air Quality and NYC Bike to test multiple prediction horizons. We are willing to add experiments on METR-LA and PEMS-BAY under the same setting here.

As can be seen in the table above, our RePST remains relatively stable and robust, demonstrating its efficacy in leveraging PLM to improve performance over longer-term predictions. For ablation studies in Figure 5, we are doing our best to conduct experiments on these two datasets. We will provide the results once we get them.

We have added the additional results of ablation studies on METR-LA and PEMS-BAY here.

Thank you again for your insightful suggestion, which has helped us strengthen our study.

Q3: What is the Projection Layer? Is it a fully connected layer? How does it project

[Response] Thank you for your valuable feedback!

The Projection Layer in RePST serves as a learnable mapping function that transforms the latent representations from the frozen PLM into the output space required for spatio-temporal forecasting. As provided in our code, the projection layer is implemented as a fully connected layer. It projects high-dimensional textual embeddings into numerical outputs that represent future spatio-temporal states. This layer bridges the gap between the PLM’s textual representations and the forecasting task, aligning the output dimensions and semantics with the requirements of spatio-temporal prediction.

Dear Reviewer LuDs,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best,

Authors

• Q2: In the original baseline papers, the MAE and RMSE for prediction horizon 12 is much lower. For instance, after full-training, Graph WaveNet has MAE 3.53 on the METR-LA dataset, and MAE 1.95 on the PEMS-BAY dataset. This limits the generalizability of the proposed method.

• Q3: Thank you for the response, I believe this should be added to the paper for better clarify.

Thanks for your prompt reply, which helps us further understand your concerns. We are pleased to hear that some of your questions have been addressed. Please find a point-to-point response to your new comments below.

[W1]

[Response] Thanks for your insightful reply. We would like to add the following references for this statement.

Li et al. [1] demonstrates that pretrained physical knowledge exists in PLMs (Pretrained Language Models) through a benchmark called VEC (Visual and Embodied Concepts), which tests the models’ understanding of visual and embodied physical concepts. The findings highlight that PLMs encode some level of physical knowledge.

Peng et al. [2] designed tasks to evaluate whether PLMs organize entities by conceptual similarities and understand conceptual properties. Their research indicates that PLMs possess some level of conceptual knowledge, which is foundational to understanding physical properties and relationships.

[1] Li L, Xu J, Dong Q, et al. Can Language Models Understand Physical Concepts?[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 11843-11861.

[2] Peng H, Wang X, Hu S, et al. COPEN: Probing Conceptual Knowledge in Pre-trained Language Models[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 5015-5035.

[W2]

[Response] PLMs often face challenges related to the vast number of word embeddings in their vocabulary, leading to the risk of overwhelming and inefficient reprogramming. To address this issue, compressing the vocabulary is an effective solution. Previous studies have employed a fully connected layer to reduce the vocabulary size [3], which integrates the original semantic meanings of the embeddings during the compression process. To be specific, it utilizes a fully connected layer, whose input dimension is the number of pretrain word embeddings, and output dimension is targeting number of compressed vocabulary.

[W3 & W4]

[Response] Thanks for your valuable feedback. Given that the decomposed sub-components are treated as features, adding too many feature dimensions is computationally inefficient. Moreover, these features are highly correlated with the target variable. Therefore, we do not recommend applying RePST to multi-dimensional spatiotemporal data. If an extension is needed, the data can be split into single-dimensional components, as demonstrated in the paper.

[Q1]

[Response] Thanks for your valuable feedback! To convert GPT-2’s word embedding into a token, we first compute the similarity (e.g., cosine similarity) between the embedding vector from our vocabulary and each embedding in the vocabulary. Identify the index of the closest match. Then, use GPT-2’s tokenizer to decode this index back into its corresponding token.

[Q2]

[Response] Thanks for your insightful feedback! We would like to clarify that our experiments were conducted in a few-shot scenario, utilizing only a limited amount of data for model training. The primary contribution of our work lies in demonstrating the effectiveness of RePST in data-scarce scenarios.

To address your concern, we have provided full training experiment results with an input and prediction length of 12 on the METR-LA and PEMS-BAY datasets. As shown in the table below, our model, RePST, performs comparably to state-of-the-art baseline models, highlighting its robustness across varying input lengths. These results underscore RePST’s capability to effectively handle shorter input sequences while maintaining competitive performance.

Additionally, our model demonstrates a significant performance enhancement of around 30% in average in short-term few-shot and zero-shot forecasting, which is the main focus of this field. This improvement highlights the superior generalization capabilities of our approach, and we believe they substantiate the significance of our contributions.

Thank you again for your insightful suggestion, which has helped us strengthen our study.