NuwaDynamics: Discovering and Updating in Causal Spatio-Temporal Modeling

Q2: Are there any generative methods in computer vision (CV) that can be integrated into data augmentation?

Yes, We believe that works related to stable diffusion and Generative Adversarial Networks (GANs), can effectively enhance data augmentation [3] [4]. We have integrated these concepts into the Related Work and Methodology sections of our paper to more comprehensively articulate the efficacy and efficiency of our methods. This integration not only underscores the relevance of stable diffusion and GANs in the context of our research but also highlights their potential in improving the robustness and performance of data-driven models.

[3] Creswell, Antonia, et al. "Generative adversarial networks: An overview." IEEE signal processing magazine 35.1 (2018): 53-65.

[4] Croitoru, Florinel-Alin, et al. "Diffusion models in vision: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

Q3: Conduct experiments to explore whether these methods remain competitive on high-resolution datasets.

In response to the your suggestion for a more comprehensive exploration of Nuwa's capabilities with high-resolution datasets, we have selected a 768x768 resolution cylinder dataset [5] to systematically validate the effectiveness of our algorithm. This dataset choice specifically addresses the performance of Nuwa in handling high-resolution scenarios, allowing us to demonstrate its robustness and accuracy in more demanding contexts.

[5] Khojasteh, Ali Rahimi, et al. "Lagrangian and Eulerian dataset of the wake downstream of a smooth cylinder at a Reynolds number equal to 3900." Data in brief 40 (2022): 107725.

Thank you for your careful and detailed review of our manuscript and for your highly appreciative evaluation of our work. Based on the your questions and recommendations, we have made many revisions that significantly improved the clarity of the paper. The new, revised paper has been uploaded to OpenReview.

Q1: whether there are clear guiding principles for environmental augmentation.

We establish a clear set of guidelines for their determination. Specifically, we control the intervention weight coefficients for environmental patches based on the weights in the attention map, and we also consider the impact of temporal decay on these coefficients, integrating both aspects into our environmental enhancement (see Equations 3-5). We have systematically deconstructed our data augmentation into two aspects to address your question more thoroughly:

Spatial perspective

we employ the Mixup technique because the state of a given region is influenced by surrounding areas, and also consider the importance of surrounding areas to assign weights to different regions. The Mixup technique is adept at extracting pertinent information from causally related neighboring regions and efficiently constructs stable regional expressions.

Temporal perspective

Inspired by studies on point processes [1-2], we postulate that the influence over time is progressively diminishing. We believe that spatio-temporal states closer to the current moment have a greater impact on future predictions. Accordingly, we have introduced a temporal decay enhancement method, which systematically reduces the number of enhancements as they approach the future moment in question.

In our analysis, we implement three distinct ablation methods. For the first strategy, A1, we mix up only the causal patches at the current time point, not considering historical patches. For the second strategy, A2, we do not account for the time decay of historical patches, instead giving equal weight to all historical data (assigning a weight of 1 to each time point). As for the third strategy, A3, we focus on the historical patches of the region itself without considering the spatial context and causal patches, adding them with decay. We organize these ablation strategies as A1, A2, and A3 respectively, and present the results (MAE) in a structured table format to facilitate a clear understanding of their impacts.

The ablation study shows that the performance significantly drops for Strategy A3, which ignores the spatial context and does not utilize causal patches for mixup. We suspect the potential reason is: if the region itself is an unimportant environmental patch, then applying a historical decay weighting to these environmental patches is meaningless. This underscores the importance of our nuwa causal reasoning. When we consider Strategy A1, which takes into account causal patches, there is a clear improvement in performance compared to Strategy A3. Finally, Strategy A2—which considers the impact of historical patches but assigns equal weight to all historical data without time decay—outperforms the other strategies, indicating that historical data plays a crucial role in predictive accuracy. Overall, none of these strategies perform as well as nuwa, which takes into account both the impact of spatial causal patches and the effect of time decay, proving that our approach is comprehensive.

For ease of understanding, we have incorporated additional ablation studies into the original manuscript to elucidate the robustness and effectiveness of our proposed methods. Furthermore, we have expanded the section on Future Work to include potential augmentation strategies.

[1] Zuo, Simiao, et al. "Transformer hawkes process." International conference on machine learning. PMLR, 2020.

[2] Mei, Hongyuan, and Jason M. Eisner. "The neural hawkes process: A neurally self-modulating multivariate point process." Advances in neural information processing systems 30 (2017).

We are deeply grateful for the comprehensive and insightful review you have provided for our manuscript, as well as for your positive feedback on our research efforts. Following your astute observations and valuable guidance, we have incorporated additional experimental evaluations with novel datasets, aiming to thoroughly respond to your queries. With these substantial improvements, we have meticulously revised the manuscript and have subsequently re-uploaded it to OpenReview for additional assessment and feedback.

W1: The paper does not seem to make it clear exactly how the process of obtaining the importance rankings in the patches is done, and it is hoped that the authors will add that detail.

In our paper, we identify key tokens within the attention map by summing and ranking the columns, pinpointing the tokens corresponding to the largest values, and designating those with the smallest values as the environmental part. Due to variations across datasets, the proportion of the environmental part slightly differs. To aid reader comprehension, we have incorporated new descriptions within the manuscript that clarify these dataset-specific differences.

W2: More discussion or explanation might have given the reader a better understanding of the consequences of using data-driven approach rather than PINN.

Data-driven approaches are favored in various fields due to their unique advantages. This approach can effectively deal with systems of high complexity and autonomously mine out new correlations from data, which greatly improves the efficiency of computation. More importantly, it reduces the reliance on deep expertise and makes the modeling process easier [1] [2] [3]. In environments where data is abundant and needs are evolving, data-driven modeling demonstrates its strong adaptability and swift predictive capabilities. Moreover, since not all contexts can be explained by physical rules, the scalability and generalizability of data-driven models make them ideal for building pervasive models.

[1] Cai, Shengze, et al. "Physics-informed neural networks (PINNs) for fluid mechanics: A review." Acta Mechanica Sinica 37.12 (2021): 1727-1738.

[2] Krishnapriyan, Aditi, et al. "Characterizing possible failure modes in physics-informed neural networks." Advances in Neural Information Processing Systems 34 (2021): 26548-26560.

[3] Cuomo, Salvatore, et al. "Scientific machine learning through physics–informed neural networks: Where we are and what’s next." Journal of Scientific Computing 92.3 (2022): 88.

Q1: Add new experiments on real pedestrian movement datasets.

We conduct experiments on the Human3.6m dataset, which features real pedestrian movement, and utilize attention maps to visualize the high-scoring regions. In these maps, we define pedestrians as the causal components. Our model adeptly identifies these causal regions. To present our findings more effectively, we include a new section in the Appendix K (Evaluations of causal discovery) dedicated to the validation of causal regions.

We express our sincere thanks for the detailed and thoughtful review of our manuscript and for the encouraging appraisal of our work. In response to your perceptive questions and constructive recommendations, we have conducted extensive ablation studies and added experimental evaluations using new datasets to comprehensively address your inquiries.

W1 & W4 : Computational complexity

Thank you for raising such insightful questions. To better elucidate the time complexity of our approach, we will systematically discuss the algorithmic complexity from two perspectives:

1. Both upstream and downstream processes are based on the transformer framework and the only difference in the data between the two is the introduction of new temporal sequences after causal interventions. Consequently, the parameters from the upstream can be directly reused by the downstream model. Here, the parameters from the upstream transformer-based model are also reused, which leads to faster convergence. Therefore, the computational burden does not increase exponentially but only adds an affordable computational load.

2. When the upstream and downstream are inconsistent, upstream self-supervised task aims to identify causal regions. Once the upstream model training is complete, it can be reused, avoiding the burden of repetitive training [1-3]. In the downstream, although the number of sequences has increased, since most sequences are essentially similar in information, the convergence speed does not lead to proportional cost increases.

We have observed that all datasets reach convergence within 100 epochs on an A100-PCIE-40GB. Since the upstream training is reusable, it only needs to be trained once and can be used repeatedly. We therefore only present the training time per epoch for the downstream (upstream is ViT). Here, we have chosen SimVP as the backbone due to its superior efficiency characteristics compared to others like PredRNN-V2.

Here, we control our sequence augmentation ratio to be nearly fivefold. We find that the efficiency does not increase by five times but only about two to three times. This further confirms that our algorithm does not cause a significant computational burden. We have added an analysis of the algorithm's complexity to the appendix to aid readers in better understanding the efficacy and efficiency of our algorithm. Additionally, we have elucidated in the model the rationale behind adopting Gaussian sampling to reduce the computational load during inference.

[1] Videogpt: Video generation using vq-vae and transformers.

[2] Generating diverse high-fidelity images with vq-vae-2.

[3]Deep learning for physical processes: Incorporating prior scientific knowledge.

W2: Lack of ablation studies to validate design choices.

Thank you for your suggestion. Indeed, based on your advice, we have included additional ablation experiments, which encompass a variety of augmentation methods and intervention sensitivity experiments (see response to Q2). We have also added an ablation study on the model's augmentation ratios, as indicated in the following table. Moreover, we have incorporated visualizations of the causal discovery regions and the augmented samples in the revised version (Appendix K) of our manuscript, furthering the systematic address of your queries.

Firstly, our augmentation ratio is determined through Gaussian sampling, which allows us to enhance the dataset to a specific proportion in a controlled manner. We have observed that different datasets exhibit slight variations in sensitivity to the augmentation ratio; however, the overall impact remains minimal. We have selected TaxiBJ+, KTH, and FireSys datasets for testing to demonstrate the sensitivity of our model to the 'number of augmented samples' parameter. Additionally, we have discovered that varying the augmentation ratios does not significantly affect the model's predictive performance, which further demonstrates the model's stability.

The results indicate that after applying Nuwa, there is a improvement in predictive performance across the board. However, taking into account the trade-off between computational overhead and performance, we have opted for a Gaussian sampling frequency with a smaller variance. This choice is aimed at better highlighting the efficiency of the model.

W3: Except for the KTH dataset, the performance gains are relatively low.

When dealing with various datasets, we adhered to the original settings presented in their respective literature and adopted the appropriate data processing strategies. Firstly, for the TaxiBJ+ dataset, despite being introduced and utilized by us for the first time, we found that normalized data yielded more stable results in our experiments. Hence, we normalized this dataset before training. For the KTH dataset, given its relatively simple characteristics as grayscale images, we followed the design strategy from [4] and did not normalize the data, allowing for a wider range of numerical variation. Additionally, for the SEVIR dataset, we conformed to the methodology of [5], did not perform normalization, and chose data from the years 2016 to 2018 for our training. For the larger-scale RainNet dataset, we implemented the design strategy from [6-7] and carried out normalization. Lastly, for the univariate physical datasets PD and FireSys, as these data were recorded in video format [8], we did not normalize but opted to proceed directly with training. We posit that the KTH dataset, with its grayscale imagery characteristics, exhibits a broad pixel range. Furthermore, due to the dataset's relatively simple feature information, there is a significant improvement in performance.

To help better understand Nuwa's capabilities, we select a 768x768 resolution cylinder dataset [9] to systematically validate the effectiveness of Nuwa. This dataset choice specifically addresses the performance of Nuwa in handling high-resolution scenarios, allowing us to demonstrate its robustness and accuracy in more demanding contexts.

[4] Recognizing human actions: a local SVM approach.

[5] Implicit Stacked Autoregressive Model for Video Prediction.

[6] RainNet v1. 0: a convolutional neural network for radar-based precipitation nowcasting.

[7]RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling.

[8] Automated discovery of fundamental variables hidden in experimental data.

[9] Lagrangian and Eulerian dataset of the wake downstream of a smooth cylinder at a Reynolds number equal to 3900.

Q1: What are the limitations of current attention map methods for identifying causal regions?

Existing methods for identifying the significance of tokens within vision transformer attention maps are all concentrated on image classification tasks. These methods [10-13] seek to ascertain the informativeness of a token by using the classification token [Cls] as a reference point. Transferring these approaches to spatio-temporal prediction tasks presents two serious issues: (1) Spatio-temporal prediction tasks, unlike classification tasks, do not require a [Cls] token. Furthermore, the final predictive information of classification task is aggregated in the [Cls] token, resulting in a lack of understanding of the individual importance of each token. (2) After summing the attention maps for each row, the total equates to one, making it challenging to identify important patches through traditional means of row averaging or summation.

In our method, we select column summation to concentrate on the aggregate attention that sub-regions direct toward a particular patch. This technique is exceedingly straightforward and universally applicable. We design and integrate it into an upstream self-supervised reconstruction task, and then we incorporate it into a causal framework. To the best of our knowledge, our method innovatively introduces attention maps to establish a causal theory within the spatio-temporal observational image data framework, marking a pioneering contribution to the field.

[10] Yuan, Li, et al. "Tokens-to-token vit: Training vision transformers from scratch on imagenet." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[11] Pan, Bowen, et al. "IA-RED $^ 2$: Interpretability-Aware Redundancy Reduction for Vision Transformers." Advances in Neural Information Processing Systems 34 (2021): 24898-24911.

[12] Rao, Yongming, et al. "Dynamicvit: Efficient vision transformers with dynamic token sparsification." Advances in neural information processing systems 34 (2021): 13937-13949.

[13] Xu, Yifan, et al. "Evo-vit: Slow-fast token evolution for dynamic vision transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 3. 2022.

The ablation study shows that the performance significantly drops for Strategy A3, which ignores the spatial context and does not utilize causal patches for mixup. We suspect the potential reason is: if the region itself is an unimportant environmental patch, then applying a historical decay weighting to these environmental patches is meaningless. This underscores the importance of our nuwa causal reasoning. When we consider Strategy A1, which takes into account causal patches, there is a clear improvement in performance compared to Strategy A3. Finally, Strategy A2—which considers the impact of historical patches but assigns equal weight to all historical data without time decay—outperforms the other strategies, indicating that historical data plays a crucial role in predictive accuracy. Overall, none of these strategies perform as well as nuwa, which takes into account both the impact of spatial causal patches and the effect of time decay, proving that our approach is comprehensive.

For ease of understanding, we have incorporated additional ablation studies into the original manuscript to elucidate the robustness and effectiveness of our proposed methods. Furthermore, we have expanded the section on Future Work to include potential augmentation strategies.

W2 & Q3: Showcase the causal discovery accuracy.

Given the absence of explicit definitions for causal and non-causal areas within the dataset, we conduct experiments on the Human3.6m dataset, a collection of real pedestrian movement data. In this context, pedestrians can be construed as regions with causal attributes, while the remaining areas can be considered as environmental. We utilize attention maps from upstream tasks to visualize significant regions, thereby highlighting our capability for causal discovery (see response to reviewer oBVm).

In these maps, we define pedestrians as the causal components. Our model adeptly identifies these causal regions. To present our findings more effectively, we include a new section in the Appendix K (Evaluations of causal discovery) dedicated to the validation of causal regions and augmentated samples.

We further claim that the assumption of causality is highly universal; it is not confined to datasets with explicit causal definitions. In addressing out-of-distribution scenarios, many studies [1-5] employ causal discovery methods to circumvent spurious correlations that may cause interference.

[1] Invariant risk minimization.

[2] The risks of invariant risk minimization.

[3] Does invariant risk minimization capture invariance?.

[4] Discovering invariant rationales for graph neural networks.

[5] Causal attention for interpretable and generalizable graph classification.

Q1: How many augmented samples are required to achieve reasonable performance?

Firstly, our augmentation ratio is determined through Gaussian sampling, which allows us to enhance the dataset to a specific proportion in a controlled manner. We have observed that different datasets exhibit slight variations in sensitivity to the augmentation ratio; however, the overall impact remains minimal. We have selected TaxiBJ+, KTH, and FireSys datasets for testing to demonstrate the sensitivity of our model to the 'number of augmented samples' parameter. Additionally, we have discovered that varying the augmentation ratios does not significantly affect the model's predictive performance, which further demonstrates the model's stability.

The results indicate that after applying Nuwa, there is a improvement in predictive performance across the board. However, taking into account the trade-off between computational overhead and performance, we have opted for a Gaussian sampling frequency with a smaller variance. This choice is aimed at better highlighting the efficiency of the model.

Q2: What if we directly perform data enhancement using the attention map?

In fact, we do use the attention map to identify environmental patches for data augmentation. We calculate the importance score of each patch by summing up the attention map column-wise. Based on these importance scores, we select environment patches in different proportions for different datasets. Then, we perform causal mixup enhancement on the environment patches, taking into account both their importance weights and historical patches.

We are currently supplementing our manuscript with additional experimental results and detailed descriptions. We are also very open to and welcome any new questions you may have.

We express our sincere thanks for the detailed and thoughtful review of our manuscript and for the encouraging appraisal of our work. In response to your perceptive questions and constructive recommendations, we have conducted extensive ablation studies and added experimental evaluations using new datasets to comprehensively address your inquiries.

W1: Add augmentation experiments.

In the field of spatio-temporal prediction, we observe that conventional data augmentation methods often lead to a decrease in predictive performance. For instance, operations like flipping or rotating images alter the physical dynamics of spatiotemporal data, thereby affecting the model's understanding of temporal continuity and dynamic changes. In contrast, data augmentation methods based on causal relationships show a significantly different effect. These methods, by deeply understanding the causal mechanisms behind the data, can effectively enhance the predictive capability of the model, thereby improving the accuracy and robustness of spatiotemporal predictions. This underscores the importance of selecting appropriate data augmentation strategies in the analysis and prediction of spatiotemporal data. B1 is ori+flip, B2 is ori+rotate, B3 is ori+VQ-VAE (which can be regarded as currently popular generative CV model). From the table, we can see that traditional data augmentation methods result in performance that is even worse than the original backbone. However, using VQ-VAE for data augmentation indeed improves the performance, but not as effectively as our proposed causality-based augmentation.

We also conduct ablations from temporal augmentation and spatial augmentation two perspective to address your question more thoroughly: In our analysis, we implement three distinct ablation methods. For the first strategy, A1, we mix up only the causal patches at the current time point, not considering historical patches. For the second strategy, A2, we do not account for the time decay of historical patches, instead giving equal weight to all historical data (assigning a weight of 1 to each time point). As for the third strategy, A3, we focus on the historical patches of the region itself without considering the spatial context and causal patches, adding them with decay. We organize these ablation strategies as A1, A2, and A3 respectively, and present the results (MAE) in a structured table format to facilitate a clear understanding of their impacts.

For the first strategy, A1, we mix up only the causal patches at the current time point, not considering historical patches. For the second strategy, A2, we do not account for the time decay of historical patches, instead giving equal weight to all historical data (assigning a weight of 1 to each time point). As for the third strategy, A3, we focus on the historical patches of the region itself without considering the spatial context and causal patches, adding them with decay. We organize these ablation strategies as A1, A2, and A3 respectively, and present the results (MAE) in a structured table format to facilitate a clear understanding of their impacts.