Spatio-temporal Twins with A Cache for Modeling Long-term System Dynamics

Q2. I really appreciate the experiments in this paper. However, the ablation study is not satisfying. The authors reported only one metric (RMSE) on only one dataset (Spherical Shallow Water). A more detailed ablation study is needed to figure out why this approach works.

A2. Thanks for your comment. Our complete ablation experiments are as follows, verifying that each component of STAC complements each other. We have corrected this in the revised manuscript.

Q3. It lacks of complexity comparison. The authors should report the parameters and FLOPs of these baseline models.

A3. Thank you for the valuable comments about comparing the complexity of our STAC framework with the baseline model. We have corrected this in the revised manuscript. The reason is that although we have a range of modules, the model depth is relatively small, which can save extensive computation costs.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. Though this paper seems to be promising, I have to say that the novelty seems to be limited. The spatio-temporal twins are actually a two-branch model. Using frequency-based approaches in the spatial domain is nothing new. The temporal module is similar to SimVP v2's [1] but with an ODE solver. The cache memory [2] is also well developed.

A1. Thank you for your comment. The novelty of this methodology comes from three parts.

• New complementary perspective, which explores spatio-temporal dynamics in both discrete and continuous manners. They are both closely related to our target long-term predictions.
• Information fusion strategy, which includes fine-grained fusion and coarse-grained fusion to tackle feature maps with different granularities.
• Cache mechanism, which aims to store previous states, thus allowing the system to remember and reuse historical data for future predictions. Compared with E3D-LSTM storing predictions at different timesteps, our cache mechanism stores feature maps with a long interval for long-term spatio-temporal predictions. Moreover, we involve more complicated interaction among current states, short-term states, and long-term states by involving short-term interaction map $A_m$ and updated maps $Q_m$. In contrast, E3D-LSTM utilizes a simple recurrent architecture, which achieves much worse performance.

Then, we describe the differences between STAC and the models mentioned in the paper [1,2].

The difference between SimVP-V2 and our STAC:

• Different objectives. SimVP-V2 focuses on standard video prediction while our STAC focuses on long-term predictions. We also generate a dataset FIRE targetting our problem.
• Different methodology. SimVP-V2 is entirely based on convolutional neural networks (CNNS) while we introduce complementary modules and cache mechanisms to benefit long-term predictions.
• Different performance. From the performance comparison, our proposed method performs much better than SimVP-V2 by over 32.18%.

The difference between LMC-Memory and our STAC:

• Different objectives. LMC-Memory focuses on video prediction while our STAC focuses on dynamics system modeling including temperature, velocity, and pressure. For example, we propose a new FIAR dataset.
• Different methodology. LMC-Memory uses an external storage device to hold motion contexts while our cache-based propagator focuses on feature maps and incorporates the interaction with short-term prediction with restoring and reusing.
• Different performance. From the performance comparison, our proposed method performs much better than LMC-Memory by over 3.84%.

Dear Reviewers,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the novelty, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

Q4 . Potential loss of spatial information in FSM: FSM applies different treatments to the same data and forcibly merges them, potentially leading to a loss of spatial information. Additionally, the direct fully connected mapping of the segmented data raises concerns about the preservation of spatial relationships and the possibility of information loss.

A4. Thanks for your comment. The information fusion of different sources is very common in recent studies [1,2]. Since we have provided extensive learnable parameters in both parts, their fusion is adaptive and can enhance semantics learning. Moreover, we conduct ablation studies as below to show that removing either part decreases the performance.

Here, we utilize fully connected mapping to make our spatial information more adaptive and more related to our target. Since we utilize an optimization process, our added parameters would be adaptively learned to provide more representation of learning capacity rather than information loss.

Ablation study, We have added four model variants as follows:

• STAC w/o FNO, which removes FNO and only uses Transformer in our FSM module.
• STAC w/o Transformer, which removes the Transformer module and only uses FNO in our FSM module.
• STAC w/o FSM, which removes FSM.
• STAC w/o FC, which removes the Fully Connected Layer.

Reference

[1] Peng, Zhiliang, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. "Conformer: Local features coupling global representations for visual recognition." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 367-376. 2021.

[2] Zhang, Jingyi, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Xiaoqin Zhang, and Shijian Lu. "DA-DETR: Domain Adaptive Detection Transformer With Information Fusion." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23787-23798. 2023.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Q3. Artificial handling and limited interpretability of IFTM: The separation of temporal and spatial processing, as well as the channel-independent merging in IFTM, appears to be a forced transformation without much interpretability. The lack of learnable factors and reliance on manual processing may hinder the scalability and extensibility of the method.

A3. Thank you for your comment. We answer it point-by-point:

1. Why separation. Compared with temporal signals, spatial signals include information from frequency domains, which cannot be considered simultaneously. Therefore, we separate the temporal and spatial processing.

2. Why transformation. We do the transformation to align the dimensions of features from different sources, which is a popular strategy like padding.

3. Why deterministic operation. Our Conv2d and FFN modules consist of extensive parameters, which allow our model with strong extensibility. Therefore, to increase efficiency, we utilize the deterministic operation for alignment. From the experiments, we can observe that our STAC performs better than baselines consistency across all datasets, which validate our scalability and extensibility.

Q2 . Limited explanation for the CRP technique: The paper mentions the use of a cache-based recurrent propagator (CRP) to prevent forgetting previous events and enhance long sequence prediction. However, it does not provide a clear explanation of the key parameter "α" and whether it is a learnable parameter. Additionally, CRP's similarity to traditional RNNs raises questions about its parallelization capabilities and potential limitations.

A2. Thanks for your comment. We solve your concerns as follows:

1. Description of parameter "α": α is a fixed hyperparameter set to $0.2$.
2. Parallelization strategy: The computational cost of our method mainly depends on the generation of feature maps, which is suitable for parallelization. Moreover, the complexity comparison of the proposed method is shown below. From the results, we can observe that the efficiency of our method is competitive.

3. When it comes to superlong prediction tasks, our cache-based design could suffer from low efficiency, which would be studied in our future works.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1 . Lack of truly innovative contributions: While the paper introduces several components and techniques, such as FSM, OTM, IFTM, CRP, Fourier-based Spectral Filters, teacher forcing, adversarial learning, and the new FIRE dataset, only CRP and IFTM can be considered as relatively novel contributions. The other techniques mentioned are already known and used in existing methods, which may limit the originality and novelty of the proposed approach.

A1. Thanks for your comment. The novelty of this methodology comes from three parts:

• New complementary perspective, which explores spatio-temporal dynamics in both discrete and continuous manners. They are both closely related to our target long-term predictions.
• Information fusion strategy, which includes fine-grained fusion and coarse-grained fusion to tackle feature maps with different granularities.
• Cache mechanism, which aims to store previous states, thus allowing the system to remember and reuse historical data for future predictions. Compared with E3D-LSTM storing predictions at different timesteps, our cache mechanism stores feature maps with a long interval for long-term spatio-temporal predictions. Moreover, we involve more complicated interaction among current states, short-term states, and long-term states by involving short-term interaction map $A_m$ and updated maps $Q_m$. In contrast, E3D-LSTM utilizes a simple recurrent architecture, which achieves much worse performance.

Then, the motivation for introducing these modules for dynamical system modeling is closely related to real-world data as follows:

• Why FSM? FSM includes FNO and Transform to explore comprehensive semantics from complementary views. On the one hand, learning from the frequency domain is crucial for our problem. For example, ERA5 datasets containing comprehensive atmospheric and ocean information need to be analyzed in the frequency domain to discern complex climate patterns. Therefore, we design FNO in the FSM module to efficiently capture these complex data in the frequency domain. On the other hand, mining Long-distance dependencies is the key to our long-term prediction tasks. Especially, in KTH datasets and FIRE datasets these dependencies are the key for accurate forecasting. With its self-attention mechanism, the Transformer effectively captures these long-distance dependencies and continuous actions in video sequences. Therefore, we design the Transformer in the FSM module to accurately identify and correlate distant frames in the video.

• Why OTM? Still, Long distance dependencies are crucial for our long-term prediction tasks. Our ODE-based module can learn these naturally without increasing the depth, i.e., infinite depth. Moreover, Turbulence data contains complex dynamic information such as fluid velocity, pressure, and temperature. Therefore, we incorporate multi-scale convolution to fully understand the physical dynamic system.

• Why IFTM? We design IFTM for information fusion from previous modules for comprehensive semantic learning. Moreover, our real datasets (e.g., SEVIR and Fire datasets) contain storm details and rapidly changing local extreme events such as flame and temperature fluctuations. Therefore, we generate informative feature maps with both coarse-grained and fine-grained semantics.

The above modules complement each other. We find from experiments that if a certain module is removed, the results will drop significantly, so we propose a unified model. Besides, this work also the contribution of new dataset and new benchmark. In the future, we will fully open-source the data sets of all working conditions and provide the code for reading and processing.

Dear Reviewers,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the contributions, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

Q3. In the part of the ablation study, some designs, for example, TF/M, CA, and SSAL, only contribute slightly improvement. However, SSAL may make the training of the framework become unstable. Others may increase the time complexity of the framework, which the authors do not report.

A3. Thanks for your comment. We have added more ablation studies on more datasets and the results are shown below. From the results, we can observe that the performance gain of our full model is between [1.27%, 14.5%] and over 5% in most cases, validating the effectiveness of every component. In addition, we include the variance of our model and STAC w/o SSAL. The results are shown below. From the results, we can observe the model is still stable comparably. Finally, we have completed experiments to verify the effectiveness of the proposed components.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. The design of the whole framework is complicated. Although the author explains the reason why they design each module, it still lacks straightforward motivation. Do such challenges really exist in the real data? This straightforward utilization of existing techniques makes the paper novelty seem incremental.

A1. Thank you for your comment. The motivation for introducing these modules for dynamical system modeling is closely related to real-world data as follows:

• Why FSM? FSM includes FNO and Transform to explore comprehensive semantics from complementary views. On the one hand, learning from the frequency domain is crucial for our problem. For example, ERA5 datasets containing comprehensive atmospheric and ocean information need to be analyzed in the frequency domain to discern complex climate patterns. Therefore, we design FNO in the FSM module to efficiently capture these complex data in the frequency domain. On the other hand, mining Long-distance dependencies is the key to our long-term prediction tasks. Especially, in KTH datasets and FIRE datasets these dependencies are the key for accurate forecasting. With its self-attention mechanism, the Transformer effectively captures these long-distance dependencies and continuous actions in video sequences. Therefore, we design the Transformer in the FSM module to accurately identify and correlate distant frames in the video.

• Why OTM? Still, long distance dependencies are crucial for our long-term prediction tasks. Our ODE-based module can learn these naturally without increasing the depth, i.e., infinite depth. Moreover, Turbulence data contains complex dynamic information such as fluid velocity, pressure, and temperature. Therefore, we incorporate multi-scale convolution to fully understand the physical dynamic system.

• Why IFTM? We design IFTM for information fusion from previous modules for comprehensive semantic learning. Moreover, our real datasets (e.g., SEVIR and Fire datasets) contain storm details and rapidly changing local extreme events such as flame and temperature fluctuations. Therefore, we generate informative feature maps with both coarse-grained and fine-grained semantics.

• Why Cache? Still, Long-term memories are crucial for our long-term prediction tasks, especially in Fire datasets. The forgetting phenomenon is common in current methods, resulting in inferior performance. Therefore, we design a cache-based mechanism to store and reuse historical feature maps.

• Why Optimization Objectives? Long-term predictions could be unreliable. Therefore, we introduce semi-supervised adversarial learning to improve the long-term predictive capacity of the model without the ground truth.

In summary, all these modules can benefit from modeling long-term dependencies for accurate long-term predictions from different perspectives (e.g., dependency mining, infinite depth, information fusion, memory reusing, and reliability).

Q2. The pictures in Figures 3, and 4 do not seem to show a significant improvement of STAC compared to other SOTA methods in terms of visualization.

A2. Thanks for your comment. We have revised the figure for better visualization. In Fig. 3, we use the red box to mark the local difference phenomenon and enlarge the area. in Fig. 4, we introduce a professional cartopy library to highlight the information on land, longitude, and latitude on the ERA5 dataset, which can better distinguish visual differences. We have updated the visualization results in Figure.3 and Figure.4 of the manuscript.

Q4. Some notations in the paper are confusing. For example, in Section 4.3, the notation definitions of input, feature map, and output are hard to match the subsequent statement.

A4. Thanks for your comment. We have introduced a notation table as below to make the paper clear.

Q5. Can authors provide the standard deviation of their experimental results?

A5. Thanks for your comment. We have added the standard deviation (SD) to the main table as shown in the table below.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Dear Reviewers,

Thank you for your invaluable feedback. As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the motivation and details, as well as the revised version of our paper. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.

Best Regards,

Authors

Q7. In the current version, they only compare STAC with video prediction baselines. How about the advanced neural operators, such as LSM [2], U-NO [3]?

A7. Thanks for your comment. We have added LSM and U-NO for a more comprehensive evaluation. The experimental results show that the STAC model outperforms LSM and U-NO on different datasets, which validates the superiority of the proposed STAC.

Q8. Are all the baselines trained by the same loss as STAC? This point is essential to ensure a fair comparison.

A8. Thanks for your comment. The loss function of the baseline follows the original paper, which is MSE loss. Considering fairness, we train STAC with only MSE loss. The experimental results are shown in the table below. It can be seen that with only MSE loss, our STAC can outperform the baselines, and adding our designed loss would further increase the performance.

Q9. More detailed ablations are expected. They should also conduct the following experiments:

Removing FNO or Transformer in FSM.

Replacing OTM with ConvLSTM or PredRNN.

Replacing the CRP with the recall gate in E3D-LSTM.

A9. Thanks for your comment. We have added five model variants as follows:

• STAC w/o FNO, which removes FNO and only uses Transformer in our FSM module.
• STAC w/o Transformer, which removes the Transformer module and only uses FNO in our FSM module.
• STAC (With ConvLSTM), which replaces the OTM module with ConvLSTM.
• STAC (With PredRNN), which replaces the OTM module with PredRNN.
• STAC (With Recall gate), which replaces the CRP with the Recall gate.

Experiments show that removing FNO and Transformer from the STAC model reduces performance especially Transformer has a significant impact. Replacing OTM with ConvLSTM or PredRNN significantly degrades performance, especially PredRNN replacement leads to memory overflow. Replacing CRP with Recall gate also shows the importance of CRP, whose removal leads to performance loss.

In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.

Q4. Are the experimental datasets temporally irregular? According to the paper, I think the input sequences are equally collected along the temporal dimension.

A4. Thanks for your comment. The dataset we use is temporally regular. The complete dataset statistics are as follows:

Table: Dataset statistics. N_tr and N_te denote the number of instances in the training and test sets. The lengths of the input and prediction sequences are I_l and O_l, respectively.

Q5. About the cache-based design. I think it is necessary to demonstrate its advancement over the temporal recall gate in E3D-LSTM.

A5. Thanks for your comment. We have added a model variant STAC w/ E3D, which replaces our cache-based design with the temporal recall gate in E3D-LSTM. The compared results are shown below. From the results, we can observe that STAC performs the best in all cases. The reason is that our module not only stores feature maps with a long interval but also involves short-term interaction maps into spatio-temporal predictions.

Q6. In addition to the performance, they should compare the efficiency with other baselines, including running time, GPU memory and parameter size.

A6. Thanks for your comment. We have added the following table as follows. From the comparison, we can observe that our efficiency is competitive. The reason is that although we have a range of modules, the model depth is relatively small, which can save extensive computation costs.

Q7. In the current version, they only compare STAC with video prediction baselines. How about the advanced neural operators, such as LSM [2], U-NO [3]?

A7. Thanks for your comment. We have added LSM and U-NO for a more comprehensive evaluation. The experimental results show that the STAC model outperforms LSM and U-NO on different datasets, which validates the superiority of the proposed STAC.

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

Q1. However, in my opinion, I think it is insufficient in novelty. STAC just combines a series of advanced models, including FNO, Vision Transformer, Neural ODE, and a similar recall mechanism proposed by E3D-LSTM. The proposed training strategy is also in a combination style. For me, it is hard to find the novel part in this model.

A1. Thanks for your comment. The novelty of this methodology comes from three parts.

• New complementary perspective, which explores spatio-temporal dynamics in both discrete and continuous manners. They are both closely related to our target long-term predictions.
• Information fusion strategy, which includes fine-grained fusion and coarse-grained fusion to tackle feature maps with different granularities.
• Cache mechanism, which aims to store previous states, thus allowing the system to remember and reuse historical data for future predictions. Compared with E3D-LSTM storing predictions at different timesteps, our cache mechanism stores feature maps with a long interval for long-term spatiotemporal predictions. Moreover, we involve more complicated interaction among current states, short-term states, and long-term states by involving short-term interaction map $A_m$ and updated maps $Q_m$. In contrast, E3D-LSTM utilizes a simple recurrent architecture, which achieves much worse performance.

Besides, this work also the contribution of new dataset and new benchmark. In the future, we will fully open-source the data sets of all working conditions and provide the code for reading and processing.

Q2. Why should they combine vision transformer and FNO? FNet has shown that the feedforward layer can perform like FFT. Why not just only use Transformer or FNO?

A2. Thanks for your comment. We have added four model variants as follows:

• STAC w/o FNO, which removes FNO and only uses Transformer in our FSM module.
• STAC w/o Transformer, which removes the Transformer module and only uses FNO in our FSM module.
• STAC w/o FNO&Transformer +Fnet, which removes the FNO and Transformer modules and replaces them with Fnet.
• Fnet, which only uses Fnet.

The compared performance is shown below. From the results, we can find that both mechanisms have a crucial effect on performance. The reason is that Transformer focuses on capturing the correlation between pixels, while FNO learns function mapping in the frequency domain. Combining them can provide a comprehensive for semantics learning. In addition, incorporating Fnet into STAC and the predictions alone cannot achieve superior performance as well.

Q3. Why can Neural ODE capture the continuous dynamics? I know that Neural ODE can achieve the adaptive depth or adaptive temporal interval. However, I think the usage here is equivalent to a simple rk4 algorithm. It is hard to claim that they learn the continuous dynamic feature. Besides, They don’t present the necessity in using Neural ODE.

A3. Thanks for your comment. Our approach utilizes Neural ODE to explore spatiotemporal relationships in a continuous way, which can allow the capture of long-range correlations naturally without increasing the depth. This point is important for effective long-term predictions. We have added a model invariant STAC w/o OTM, which removes the ODE module. The performance is shown below. From the results, we can observe our ODE module is necessary for accurate long-term prediction.

Thanks for your feedback and we are happy to resolve your further concerns as follows:

Q1. I don't think the usage in STAC is Neural ODE. It is just a rk4 algorithm. If you do want to claim that your model takes benefits from Neural ODE, you need to use the learnable temporal steps $t$.

A1. Thanks for your comment. Clearly, the earliest neural ODE [1] also focuses on the fixed depth, but introduces the continuous propagation with benefits. Moreover, [2] also utilizes the fixed step $t$ with performance improvement, which shows the benefit of adopting neural ODE with fixed steps. Moreover, we would include learnable $t$ in our future work.

Would you please clarify this comment "which can allow the capture of long-range correlations naturally without increasing the depth" in your rebuttal? What kind of long-range correlations do you mean?

A2. Thanks for your comment. Here, the long-range correlations mean the relationships of pixels far away in tensor $\bar{I}^{in}$, which require a large receptive field to capture. Here, neural ODE utilizes continuous propagation to get rid of stacking a range of CNN blocks.

Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.

[1] Chen et al. Neural Ordinary Differential Equations.

[2] Jin et al., Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs

Dear Reviewer rchA,

Thank you very much again for the time and effort put into reviewing our paper. We believe that we have addressed all your concerns in our response. We have also followed your suggestion to improve our paper and have added additional experimental analysis. We kindly remind you that we are approaching the end of the discussion period. We would love to know if there is any further concern, additional experiments, suggestions, or feedback, we kindly hope that you can consider increasing the score.

Best regards,

All authors

Thanks for your feedback and we are happy to resolve your further concerns as follows:

Q1. The description of "which can allow the capture of long-range correlations naturally without increasing the depth" is still not well-supported.

A1. Thanks for your comment. Here, we revised our statement into "which can allow the capture of long-range correlations naturally with robustness" since Neural ODE has the property of robustness with increased propagation depths, as stated in Sec. 5.4 of [1]. This is one of the most important properties of Neural ODE compared with discrete methods. We have revised the manuscript accordingly.

[1] Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs

Q2. About the soundness.

A2. Thanks for your comment. We have added extensive clarification and experiments to address all the concerns. Moreover, the other reviewers all give a "3" for soundness. If there are any remaining issues or if you require further clarification, please feel free to inform us.

Q3. About the novelty.

A3. Thanks for your comment. As mentioned earlier, our method consists of three parts, i.e., complementary perspective, information fusion strategy and cache mechanism, which are novel compared with existing works. This work also the contribution of new dataset and new benchmark in advancing the community. And we are committed to making our complete code and the dataset publicly available.

As the deadline for the author-reviewer discussion phase is approaching, we hope to make sure that our response sufficiently addressed your concerns regarding the revised version. We hope this could align with your expectations and positively influence the score. Please do not hesitate to let us know if you need any clarification or have additional suggestions.