High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representation

[W2] Qualitative results for reconstruction stage

A: Thank you for your constructive feedback. We appreciate your suggestion regarding the inclusion of visualization results. We have added qualitative visualization results for the reconstruction stage in the main paper. Additionally, we have included comprehensive visual comparisons in the supplementary material to further validate the performance improvements claimed in the paper.

These visualizations clearly demonstrate that our proposed reconstruction method preserves significantly more details and achieves consistent accuracy across all frames. In contrast, other methods exhibit noticeable deviations in the reconstructed structure by the final frame, such as incorrect patterns or blurred regions. These qualitative results further validate the dramatic performance improvements claimed in the paper.

[W3] Grammar and Writing Issues

A: Thank you for your careful review. We have thoroughly rechecked the entire manuscript and corrected grammatical errors to improve the overall readability and quality of the paper.

Thank you for your detailed and constructive feedback. We greatly appreciate your insights, which have helped us identify areas for improvement in our work.

[W1] Performance gap in reconstruction

A: We appreciate the reviewer’s observation regarding the large performance gap between our method and the baselines in Table 1. To clarify, in our original experiments, we ensured fair comparisons by aligning all methods under a unified reconstruction setting. Specifically, we allowed all parameters to be freely optimized, as in our proposed method. However, baseline methods were originally designed with some parameters fixed (e.g., RGB or opacity), which serve as an implicit anchor to guide the alignment of Gaussians with the dynamic changes of the reconstruction target. This guidance mechanism is absent in our unified setting, inherently making the reconstruction task more challenging.

Without this guidance, optimizers may struggle to identify which parameters to adjust for better dynamic reconstruction. This leads to convergence issues. To further investigate this phenomenon, we conducted additional experiments where we compared the baselines both in their original settings (with fixed parameters) and in our unified setting. The updated results are summarized below:

As demonstrated, when the baselines adhere to their original settings, the performance gap narrows significantly. However, in the unified setting, baseline methods face convergence issues due to the increased complexity of the optimization task. Our proposed strategy efficiently handles these challenges, achieving superior performance.

Regarding the suggestion to include baselines outside the 3DGS family: recent studies have consistently demonstrated that 3D Gaussian-based methods outperform earlier approaches such as NeRF and voxel grid representations, particularly in dynamic scene reconstruction tasks. A key distinction lies in the nature of these methodologies: 3D Gaussian-based techniques focus on explicit reconstructions, providing a direct and interpretable representation of the scene, while methods like NeRF and voxel grids adopt implicit reconstruction frameworks, relying on latent representations that are less suited for explicit scene modeling and manipulation. Incorporating baselines from outside the 3DGS family would introduce discrepancies in objectives and compatibility, potentially diluting the focus of our study.

In the context of our work, which is designed to leverage the advantages of 3D Gaussian representations for both reconstruction and subsequent prediction. This experimental design aims to emphasize the effectiveness of our reconstruction strategy in challenging scenarios based on 3DGS techniques. Demonstrating improvements over existing 3DGS methods highlights the robustness and adaptability of our approach.

We hope this clarification and additional experimental evidence address the reviewer’s concerns. Thank you again for the constructive feedback.

[Q3] Limitations of our method

A: Thank you for emphasizing the importance of discussing the limitations of our method.

• From the application perspective, while our approach demonstrates strong performance in reconstructing and predicting highly dynamic radar sequences, its current scope is somewhat specialized, focusing primarily on radar-based weather nowcasting and sequence prediction. We acknowledge that broader applications, such as general 3D scene reconstruction and dynamic modeling for robotics or AR/VR environments, have not been explored in this work. Expanding our method into a foundational framework for 3D world modeling is a direction we are actively pursuing.

• From the technical perspective, the reconstruction strategy employed in our method, while effective, is somewhat tailored to radar sequences, which may limit its adaptability to other dynamic 3D scenarios, such as autonomous driving, robotic movements or manipulation. To address this, we are working toward developing a more generalized reconstruction framework and simplifying the pipeline to enhance its versatility and computational efficiency across diverse domains.

We'll add a short discussion of limitations to the main paper.

[Q1] A fixed number of Gaussian primitives

A: Thanks for your detailed question. Yes, the number of Gaussian primitives is fixed across all frames in our model. This setting is motivated by the necessity to ensure spatiotemporal consistency during sequence reconstruction and reparameterization. Dynamically adding or removing Gaussian primitives would disrupt this consistency, and make it challenging for the model to capture the temporal evolution of each individual 3D Gaussian.

Our approach aligns with findings in Taming 3DGS [1] which highlights that the densification operations in the original 3D Gaussian Splatting (3DGS) framework may introduce challenges for subsequent training. Similarly, many studies [2-7] incorporating 3D Gaussians into deep learning frameworks also adopt a fixed number of primitives for training simplicity and stability.

In our implementation, we set the number of Gaussian primitives to $3 \times 2^{14}$, balancing computational efficiency and reconstruction precision. During initialization, these primitives are randomly distributed within regions containing valid radar echoes. To ensure their effectiveness, we propose a Bidirectional Reconstruction Scheme coupled with local detail and global trend constraints, which enables each primitive to meaningfully contribute to the reconstruction process and remain aligned with the motion of the corresponding cloud structures.

We appreciate your feedback and hope this explanation addresses your concerns.

Refereces:

[1] Mallick, Saswat Subhajyoti, et al. "Taming 3dgs: High-quality radiance fields with limited resources." arXiv preprint arXiv:2406.15643 (2024).

[2] Shen, Qiuhong, et al. "Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction." arXiv preprint arXiv:2403.18795 (2024).

[3] Yi, Xuanyu, et al. "MVGamba: Unify 3D Content Generation as State Space Sequence Modeling." arXiv preprint arXiv:2406.06367 (2024).

[4] Ziwen, Chen, et al. "Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats." arXiv preprint arXiv:2410.12781 (2024).

[5] Zhang, Kai, et al. "Gs-lrm: Large reconstruction model for 3d gaussian splatting." European Conference on Computer Vision. Springer, Cham, 2025.

[6] Tang, Jiaxiang, et al. "Lgm: Large multi-view gaussian model for high-resolution 3d content creation." European Conference on Computer Vision. Springer, Cham, 2025.

[7] Lu, Guanxing, et al. "Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation." European Conference on Computer Vision. Springer, Cham, 2025.

[Q2] GauMamba's memory efficiency

A: Thank you for raising this insightful question. The observed discrepancy in memory efficiency at different resolutions arises from the fundamental differences between the underlying mechanisms of the baseline methods and our GauMamba model.

The baseline methods rely on convolutional architectures, where memory usage is directly proportional to the size of the feature maps. This results in quadratic growth in memory consumption as the spatial resolution increases (considering horizontal dimensions). In contrast, our approach employs a fixed number of Gaussian primitives, making memory usage independent of the resolution. Instead, memory consumption in our model scales linearly with the number of Gaussian primitives.

At lower resolutions, most radar echo details are lost, which can significantly reduce the practical utility of the predictions. For example, many intense meteorological events evolve from small-scale structures, and low-resolution models may fail to capture these early-stage developments. Additionally, low-resolution predictions can overly average regions of high reflectivity, obscuring critical localized features. High-resolution predictions, on the other hand, provide more detailed guidance for disaster prevention and mitigation, enabling timely and precise decision-making.

Furthermore, the number of Gaussian primitives in our method is tailored to the specific resolution of the radar data. Reconstructing low-resolution radar sequences requires fewer Gaussian primitives to represent the underlying features. Therefore, at low resolutions, the reduced number of Gaussian primitives also leads to higher memory efficiency.

In Figure 4, we aimed to highlight that under the current experimental settings, our approach demonstrates superior memory efficiency, particularly at higher resolutions where practical applications are most relevant.

[W2] Dataset size and diversity

A: Thank you for raising this important concern. Our experiments were conducted on two 3D radar sequence datasets: MOSAIC and NEXRAD.

• Dataset diversity:

  MOSAIC consists of a full year of radar observations from the region spanning the intersection of Northeast Asia and Southeast Asia, near the western Pacific. This region exhibits significant climatic diversity due to its unique geographical and meteorological conditions. It experiences a wide range of weather phenomena such as convective rain, stratiform precipitation, monsoons, and typhoons throughout the year, making it an ideal dataset to capture various weather dynamics.

  NEXRAD comprises radar data collected from multiple observation stations across the United States. This dataset reflects not only climatic diversity but also geographical variability, as it covers regions including such as mountains, plains, and urban areas.

  Such variability enhances the robustness of our method's evaluation.

• Dataset size:

  We acknowledge the broader challenge of limited availability of diverse 3D radar datasets. Several factors contribute to this limitation:

  • Technical constraints: Many radar systems cannot reliably capture accurate 3D observations.

  • Research focus: Current meteorological studies often prioritize 2D low-altitude radar imagery, with the collection and utilization of 3D radar data still in exploratory stages.

  • Data access restrictions: Stringent security and privacy policies in many countries prevent public sharing of 3D radar data.

  Despite these challenges, the NEXRAD dataset, derived from the U.S. NEXRAD WSR-88D radar network, is widely regarded in the remote sensing community for its reliability and representativeness. Moreover, the combination of NEXRAD with MOSAIC dataset introduces complementary perspectives, as these datasets stem from distinct meteorological systems and utilize different radar equipment, enhancing the diversity of our experimental setup.

  To further address concerns about dataset size, we are extending our experiments by incorporating additional radar data from 2020 and 2021. This expansion aims to evaluate whether increasing the dataset size further enhances model accuracy. As the experiment is being conducted, we plan to update the results in the next few days.

  Additionally, we plan to publicly release this three-year dataset to support future research in the community.

Thank you for your valuable feedback and thoughtful suggestions. We appreciate the opportunity to improve our paper based on your comments. Below, we provide detailed responses to each point raised.

[W1] Review and compare with previous Mambda + Gaussian methods

A: We sincerely thank the reviewer for the valuable suggestion. To address this, we plan to expand the Related Work section by briefly reviewing existing methods that combine Mamba and Gaussian models and highlighting the distinctions of our proposed GauMamba method. Specifically, we will add the following content to Section 2.1:

Several studies [1-4] have attempted to integrate Mamba or Transformer architectures with 3D Gaussian. However, these methods primarily focus on reconstructing 3D Gaussians from single or multi-view images. In contrast, our method emphasizes utilizing sequences of 3D Gaussians to represent the evolution of 3D radar echo sequences, and employing the GauMamba model to predict future frames. Existing methods lack the capacity to retain past observations. In contrast, our proposed Memory-Augmented GauMamba effectively incorporates observations from preceding frames to model the spatiotemporal evolution of 3D Gaussians, significantly improving prediction accuracy.

Additionally, we have included new comparisons in the Experiment section with the original Mamba-based model as a baseline. The results are presented in the following table:

Table 1 Experiment results in MOSAIC:

Table 2 Experiment results in NEXRAD:

We would like to emphasize that the original Mamba model lacks memory capabilities, meaning it cannot retain information from previous frames and can only predict the next frame based on the current state. The comparison with GauMamba clearly demonstrates that the original Mamba performs suboptimally in spatiotemporal sequence tasks.

More importantly, the approaches and experiements developed within our proposed framework, which reformulates the 3D radar sequence prediction task by first re-representing the sequence with 3D Gaussians and then predicting future frames, significantly outperform traditional methods. This underscores the effectiveness of our novel framework. Additionally, our Memory-Augmented Mamba Predictive Model further enhances the predictive power of Mamba-based models.

References:

[1] Shen, Qiuhong, et al. "Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction." arXiv preprint arXiv:2403.18795 (2024).

[2] Yi, Xuanyu, et al. "MVGamba: Unify 3D Content Generation as State Space Sequence Modeling." arXiv preprint arXiv:2406.06367 (2024).

[3] Ziwen, Chen, et al. "Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats." arXiv preprint arXiv:2410.12781 (2024).

[4] Zhang, Kai, et al. "Gs-lrm: Large reconstruction model for 3d gaussian splatting." European Conference on Computer Vision. Springer, Cham, 2025.

I really appreciate the comprehensive explanation and additional comparison with Mamba. My concerns have been sufficiently addressed, so I raised my score to 8 (accept, good paper). I encourage the authors to consider making their dataset publicly available in the future, as it would significantly benefit the fields of weather forecasting and 4D reconstruction.

Thank you for your insightful and constructive feedback. We greatly appreciate your comments and have carefully addressed each point in our detailed responses below.

[W1] A short discussion of the proposed dataset

A: Thank you for your valuable suggestion. Including a brief discussion about the two dataset in the main paper will benefit readers. We will add the following content to Section 4.1:

The datasets used in this study include NEXRAD and MOSAIC. NEXRAD comprises 6255 radar observations of severe storms in the U.S., with 3D reflectivity data sampled at 5-minute intervals and a resolution of $44 \times 512 \times 512$. Seven radar features, such as reflectivity, azimuthal shear, differential reflectivity, and so on, are included. MOSAIC records 24,542 radar observations of storms in Guangdong, China, with 6-minute intervals and a resolution of $38 \times 384 \times 512$, focusing solely on intensity data of radar echoes. Both datasets are preprocessed to ensure consistent vertical spacing and are divided into training, validation, and test sets. The prediction task involves forecasting up to 20 future frames based on 5 observed frames. For further information, please refer to the supplementary material C.1.

[W2] Clarify the public plan for the dataset

A: Thank you for your question regarding the dataset. We confirm that we plan to make the datasets used in this work publicly available. Specifically, this paper utilizes two datasets: NEXRAD and MOSAIC .

• NEXRAD: This dataset is derived from the U.S. NEXRAD WSR-88D radar network. We have already uploaded the processed dataset to Hugging Face for public access, and it can be accessed via the following link: https://huggingface.co/datasets/Ziyeeee/3D-NEXRAD. Additionally, we are currently extending this dataset to include data from 2020 and 2021. These years are undergoing final organization and processing and are expected to be released in this month (November 2024).

• MOSAIC: This dataset originates from the National Meteorological Centre. Its release requires additional approvals and security reviews. We are actively coordinating with the relevant authorities to expedite the process and will make the dataset available as soon as possible.

[W3] The evaluator model of LPIPS and a pretrained model with radar data

A: Thank you for your insightful question. We employed a pretrained AlexNet model for the LPIPS evaluation, following the settings in the original 3D Gaussian Splatting (3DGS) paper [1] and the suggestion in LPIPS framework [2]. We will clarify this detail in the revised manuscript.

Regarding your concern about whether a model pretrained on a general-purpose dataset is suitable for radar data, we agree that this is an interesting and important question. To address this, we conducted additional experiments. One notable challenge in radar data lies in the scarcity of labeled datasets for supervised training. However, as highlighted in [2], self-supervised models like BiGAN and supervised AlexNet calibrated with human perceptual judgments achieve comparable performance in measuring perceptual distance (68.4 vs. 69.8 2AFC scores). This indicates that the evaluator model does not necessarily need to be trained on a classification task to perform effectively; self-supervised models can achieve results on par with supervised ones.

Based on this observation, we pretrained a BiGAN model on radar data in a self-supervised manner and used its encoder as the evaluator for LPIPS. This approach resulted in the radar-specific perceptual metric, $\text{LPIPS}_{\text{Radar}}$, as shown in the updated tables below:

Table 1 Experiment results in MOSAIC:

Table 2 Experiment results in NEXRAD:

The results demonstrate that LPIPS_Radar, derived from the BiGAN evaluator, is well-aligned with the original LPIPS results. More importantly, it highlights perceptual differences that were previously undetected by the original LPIPS, such as a difference between 0.126 and 0.129 in LPIPS, but a more significant difference between 0.665 and 0.699 in LPIPS_Radar.

Notably, in Table 1, the scores for ConvGRU and DiffCast show discrepancies between LPIPS and LPIPS_Radar. Referring to Figure 6 left, it can be clearly observed that ConvGRU fails to predict the next few frames accurately, instead providing a smoothed average result. In contrast, DiffCast produces results that are closer to the ground truth but with some noise. The higher LPIPS score for DiffCast indicates that LPIPS lacks robustness to noise in radar data, while LPIPS_Radar more accurately reflects the perceptual differences between the two methods. This tailored evaluator not only better suits the unique characteristics of radar data but also enhances the robustness of our evaluation. This new method provides a more accurate perception of radar data and its model predictions.

Additionally, in response to Reviewer yyVm’s suggestion, we have included a new comparison with the original Mamba model in the updated table, highlighting the improvements brought by our GauMamba model.

We appreciate your valuable suggestions and will incorporate these results and discussions into the revised manuscript.

(Continuation from above)

References:

[1] Kerbl, Bernhard, et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Trans. Graph. 42.4 (2023): 139-1.

[2] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[W4] Minor visual artifacts in Figure 2

A: Thanks for your suggestion to improve clarity. The semi-transparent design was originally intended to illustrate that the data for the $t+1$ frame flows through the network in the same manner as the $t$ frame. However, we realize that this design might cause confusion.

To address this, we have removed the semi-transparent elements. Instead, we have provided a clear explanation in the figure caption.