STORM: Spatio-TempOral Reconstruction Model For Large-Scale Outdoor Scenes

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the strengths of our work, and we address your concerns and questions in detail below. We hope these clarifications demonstrate the robustness and versatility of STORM.

Q1: Generalizability Beyond the Waymo Open Dataset

A: We have addressed this point in detail in the general response. In summary, we evaluated STORM on two additional datasets—NuScenes and Argoverse2—to validate its generalizability. These results, included in the revised manuscript, demonstrate STORM’s robustness and versatility across diverse real-world scenarios. For further details and results, please refer to the general response.

Q2: Robustness to Sparse Observations

A: This is a very insightful and inspiring suggestion that motivates us to explore more. We believe showing this study will further highlight the flexibility of our approach. Now in the revised manuscript, we investigated STORM's robustness to varying numbers of input timesteps. Leveraging the sequence-to-sequence nature of our Transformer-based model, STORM can flexibly adapt to different numbers of input timesteps during both training and testing. We conducted two ablation studies (detailed in Fig. B.2):

• (1) Testing STORM trained with 4 input timesteps on different number of test-time input timesteps (e.g. 1, 2, 6, 10) without re-training (zero-shot transfer). Results show that STORM generalizes well to unseen configurations, with peak performance achieved at 4 timesteps, as expected.
• (2) Retraining STORM with different number of input timesteps (including 1, 2, 4, 6, 10). Results indicate that increasing timesteps improves performance. Notably, when trained with a single timestep, STORM generalizes to a future prediction framework (by being able to predict Gaussian velocities given a single timestep), achieving a remarkable 2–4 PSNR improvement on dynamic regions compared to per-scene optimization methods using 4 timesteps. These ablations highlight STORM’s flexibility and robustness under varying conditions, further validating its applicability to real-world scenarios.

These ablation studies highlight STORM’s robustness to sparse observations and varying training time and test-time input timesteps. For additional details, please refer to the revised manuscript.

We hope these updates and clarifications address your concerns and further demonstrate the robustness, versatility, and real-world applicability of STORM.

Thank you again for your valuable feedback, which has significantly improved the manuscript!

Please let us know if there are additional points we can address.

Dear Reviewer HXs2,

Thank you for your thoughtful and constructive review of our submission. Your feedback has been instrumental in strengthening our work, particularly regarding generalizability and robustness under sparse observations.

With the discussion deadline approaching, we want to ensure that all your concerns are fully addressed. Please do not hesitate to reach out if you require further clarifications or analyses.

To address your concerns:

• We conducted experiments on NuScenes and Argoverse2, demonstrating STORM’s applicability across datasets.
• We included new ablation studies that explore performance under varying numbers of input timesteps, highlighting robustness to sparse observations and different training/testing configurations. We highly appreciate your insightful review of this point!

We believe these updates address your concerns and would greatly appreciate it if you could review the revisions. Your confirmation or additional feedback would be highly valued.

Additionally, we hope you could consider raising your score based on our rebuttal, as we believe these revisions, guided by your thorough review, have greatly improved the paper.

Thank you again for your time and valuable suggestions!

Best regards,

Authors

We thank the reviewer for the detailed review and thoughtful feedback. We have incorporated the suggestions into the revised manuscript and addressed each of the raised questions and concerns below. We hope these clarifications effectively demonstrate the robustness and contributions of our work.

Q1: Generalizability Across Multiple Datasets

A: We have addressed this point in detail in the general response. In summary, we evaluated STORM on two additional datasets—NuScenes and Argoverse2—to validate its generalizability. These results, included in the revised manuscript, demonstrate STORM’s robustness and versatility across diverse real-world scenarios. Please refer to the general response for further details and results.

Q2: Dynamic Scene Reconstruction Focus

A: We acknowledge the relevance of prior works, such as VastGaussian [1] and Dynamic 3D Gaussian Fields [2], and have added references to these methods in the revised manuscript. Below, we explain their relationship to our work:

• VastGaussian [1]: This work focuses on improving the memory efficiency and scalability of per-scene optimized 3D Gaussian Splatting. While this is orthogonal to our goal of real-time, feed-forward dynamic reconstruction, we believe STORM can benefit from the techniques proposed in VastGaussian in future research.
• Dynamic 3D Gaussian Fields [2]: This method leverages bounding box annotations to learn dynamic objects. Since STORM does not use label information, we have focused on comparing state-of-the-art methods that similarly do not require annotations. For this reason, we have not included this approach in our current baselines.

Title Clarification：Regarding the title, while we initially emphasized the temporal aspect of reconstruction, we agree that explicitly including “dynamic” in the title would better highlight STORM’s focus and will consider this revision.

Q3: Ablation Studies on Proposed Components

Thank you for highlighting the need for ablation studies. We have conducted detailed ablation analyses on key components, including velocity regularization, motion token count, and input timesteps. For a full summary of these results, please refer to the general response. These studies further validate STORM’s robustness and the effectiveness of its design choices.

We hope these updates and clarifications address your concerns and demonstrate STORM’s strengths, versatility, and real-world applicability. Thank you again for your valuable feedback, which has greatly improved our work. Please let us know if there are any additional points we can address.

[1] Lin, Jiaqi, et al. "VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction." CVPR 2024. [2] Fischer, Tobias, et al. "Dynamic 3D Gaussian Fields for Urban Areas." arXiv preprint arXiv:2406.03175 (2024).

Dear Reviewer N7nA,

Thank you for your detailed and thoughtful review of our submission. Your feedback on generalizability, comparisons, and ablation studies has been immensely helpful in refining our work.

With the discussion deadline approaching, we want to ensure that all your concerns are fully addressed. Please do not hesitate to reach out if you require further clarifications or analyses.

In our rebuttal and revised manuscript, we have addressed your points:

• Generalizability has been demonstrated with additional experiments on NuScenes and Argoverse2, where STORM achieved state-of-the-art performance again, validating STORM’s robustness across diverse datasets.
• References to related works, such as VastGaussian and Dynamic 3D Gaussian Fields, have been added for context.
• Comprehensive ablation studies now evaluate the impact of motion tokens, velocity regularization, and other components.

We hope these updates and clarifications have addressed your concerns and strengthened our submission. If you have had a chance to review our revisions, we would greatly appreciate your feedback or confirmation on whether the updated manuscript meets your expectations. Your confirmation or additional comments would mean a great deal to us.

Additionally, we hope you could consider raising your score based on our rebuttal, as we believe these revisions, guided by your thorough review, have greatly improved the paper.

Thank you once again for your time, thoughtful feedback, and support.

Best regards,

Authors

We thank the reviewer for the thoughtful review and constructive comments. We address the questions and concerns below and have incorporated additional clarifications and updates into the revised manuscript. We hope that our revisions and this rebuttal address the reviewer's concerns.

Q1. Short video sequence v.s. Long video sequence.

A:

• Computation efficiency. We would like to emphasize the computational efficiency of STORM. Applying STORM to a 20-second video sequence requires approximately 2 seconds of processing time, compared to hours of training time required by per-scene optimization methods. This represents a significant improvement in efficiency. STORM is already capable of handling 64 or more input views during training with our current implementation.
  • To further address your concerns, we trained a STORM variant with 10 timesteps × 3 views (30 input views in total) spanning a 10-second duration (views evenly spaced at 1-second intervals). This configuration achieves scene coverage up to 100m. A video demo illustrating this capability is available here: https://anonymousi079j.github.io/STORM_review/assets/videos/10s_demo.mp4. Note that we did not change any hyperparameters for this variant so this example is purely for illustration.
  • Additionally, STORM is expected to benefit from advanced techniques such as FlashAttentionV3, building on our current implementation, which utilizes FlashAttentionV2. These advancements could further extend our model's context length and scalability.
• Autonomous vehicles (AV) application. For AV scenarios, long rollouts (e.g., 20 seconds) are rarely necessary for training or testing. Typically, adversarial scenarios in AV testing require shorter sequences (e.g., 4-5 seconds). STORM’s design aligns well with these practical requirements, making it a compelling fit for AV applications.

Q2. Quantitative ablation. Thank you for emphasizing the importance of quantitative ablations. We have conducted comprehensive ablation studies analyzing the contributions of key components, including velocity regularization, motion token count, and input timesteps. These findings are summarized in the general response and detailed in the revised manuscript (Line 429-460 and Appendix B.2 (Figures B.1–B.2)). Please refer to the general response for a full overview of these results, which highlights STORM’s robustness and flexibility under varying configurations.

Q3. Side-by-side visual comparison. We appreciate this suggestion to include side-by-side comparisons. To address this, we have updated Figure 3 to include a pointer to our project page, where readers can access side-by-side video comparisons (see the “Human Modeling with Latent-STORM” section). Due to space constraints, ground truth images were not included in the manuscript but can be added in the camera-ready version if space permits.

Q4: Reference to Handling Exposure Mismatches with Affine Transformation The revised manuscript now references the "Urban Radiance Fields" [1] work, which employs learnable affine transformations to align exposure differences across cameras in a per-scene optimization setting. This provides additional context for our approach.

We hope these updates and clarifications address your concerns and demonstrate STORM’s strengths, versatility, and real-world applicability. Thank you again for your valuable feedback, which has greatly improved our work. Please let us know if there are any additional points we can address.

[1] Rematas, Konstantinos, et al. "Urban radiance fields." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Dear Reviewer JKrr,

Thank you for your constructive feedback on our submission. Your comments on long sequences, ablation studies, and visualization have been instrumental in refining the manuscript.

With the discussion deadline approaching, we want to ensure that all your concerns are fully addressed. Please do not hesitate to reach out if you require further clarifications or analyses.

To address your points:

• We expanded our experiments to demonstrate STORM's scalability to more input frames or longer sequences, with new video demos included on the project page.
• Detailed ablation studies have been added to validate the contributions of individual components.
• Visualizations now include a link to a side-by-side video comparison for STORM v.s. Latent-STORM on human modeling.

We hope these updates address your concerns. If you have had a chance to review our revisions, we would greatly appreciate your feedback or confirmation on whether the updated manuscript meets your expectations.

Additionally, we hope you could consider raising your score based on our rebuttal, as we believe these revisions, guided by your thorough review, have greatly improved the paper.

Thank you once again for your time and insights.

Best regards,

Authors

Unfortunately the authors have not produced any clarifications of the points addressed by reviewers. In this situation, I'm inclined to reject this paper which, while interesting, has a number of issues that remain unanswered.

We sincerely thank the reviewer for the insightful comments. We have incorporated these suggestions into our revised manuscript and provided detailed explanations for each of the reviewer's questions below. We hope that our revisions and this rebuttal address the reviewer's concerns.

In addition, we will open-source our code for the community to examine the implementation details more closely.

Q1: How are the images $\hat{I}$ computed for a given training batch based on aggregation and transformation?

A: Thank you for highlighting the need for clarity. We apologize for the lack of clarity in our original manuscript and appreciate the opportunity to clarify these. Here is a step-by-step explanation:

1. Input Batch Structure: We describe the model’s behavior from a batch perspective.

• Context timesteps ($t$): These are the input timesteps from which we predict per-frame 3D Gaussians (3DGS) and velocities. For four context frames, $t$ has the shape $(b, \text{ctx}_t)$, where $b$ is the batch size and $\text{ctx}_t = 4$.
• Target timesteps ($t'$): These are the timesteps at which we render $\hat{I}$, with shape $(b, \text{tgt}_t)$.

2. Computing Time Differences: We calculate dense time differences between target and context timesteps: $\text{tdiff} = t' - t$, resulting in a tensor of shape $(b, \\text{tgt}_t, \\text{ctx}_t)$. The pseudo-code is tdiff = tgt_time.unsqueeze(-1) - ctx_time.unsqueeze(-2). This corresponds to the $( t' - t)$ term in Equation (1) of our paper.

3. Preparing Velocity and Center Tensors:

• Velocities ($\mathbf{v}_t$): Predicted for each context frame, shape $(b, \text{ctx}_t, N, 6)$, where $N= v\times H \times W$ is the number of 3D GS at each timestep, and $v, H, W$ denotes the number of camera views, and the height and width of input images.

• Centers ($\boldsymbol{\mu}_t$): Gaussians' centers, shape $(b, \text{ctx}_t, N, 3)$.

• We expand these tensors to align with target timesteps: $\mathbf{v}_t \rightarrow (b, \text{tgt}_t, \text{ctx}_t, N, 6)$ and $\boldsymbol{\mu}_t \rightarrow (b, \text{tgt}_t, \text{ctx}_t, N, 3)$

4. Applying the Transformation:

• With all tensors now sharing the shape $(b, \text{tgt}_t, \text{ctx}_t, \ldots)$, we compute Equation (1) using element-wise operations.
• After applying Equation (1), we obtain the transformed centers $\boldsymbol{\mu}_{t' \rightarrow t}$ with shape $(b, \text{tgt}_t, \text{ctx}_t, N, 3)$. We simplify this notation to $\boldsymbol{\mu}'_t$ because OpenReview suffers from formula display issues when using excessive subscripts.

5. We aggregate context to form the amodal representation: $\mathcal{G}_{t'} = \bigcup\_t \boldsymbol{\mu}'_t$, resulting in a shape of $(b, \text{tgt}_t, \text{ctx}_t \times N, 3)$. Here, the third dimension $\text{ctx}_t \times N$ corresponds to the "amodal" representation we define in the manuscript, which combines all Gaussians from different context frames into a single set for each target timestep.

6. We render $\hat{I}_{t'}$ at each target timestep $t'$ by projecting the aggregated Gaussians at this timestep (i.e., selecting one sample from the second dimension of $(b, \text{tgt}_t, \text{ctx}_t \times N, 3)$) onto the image plane of the desired viewpoint.

Q2: What are the shapes of $q$ and $k$ in Equation (4), and how are the weights used for generating the motion segmentation visualizations?

A:

• Shapes of $q, k$: We apologize for the ambiguity in the original manuscript. The updated paper now includes additional clarifications (Lines 232–253). In brief:

  • $q \in \mathbb{R}^{M \times e'}$ and $k \in \mathbb{R}^{H \times W \times e'}$, where $M$ is the number of motion tokens, $H$ and $W$ are the height and width of the input images, $e’=32$ is a low-dimensional motion embedding space (mentioned in Line 916) used to measure the dot-product similarity between queries and keys.
  • For further details, we strongly encourage the reviewer to refer to the updated manuscript, which provides a clearer and more comprehensive explanation.

• Motion Segmentation Visualization: In the updated Equation (4), we clarify that the motion assignment weights $\mathbf{w} \in \mathbb{R}_{+}^{H \times W \times M}$ are in the range $(0, 1)$. In the updated Lines 504–506, we clarify that the motion segmentation mask is derived by applying an argmax operation on the per-pixel assignment weights $\mathbf{w}$ along the motion token dimension $M$.

• Scene Background: Thank you for raising this insightful point. To clarify, we do not explicitly differentiate between background and foreground. Static regions may be over-segmented into separate groups since the assignment process is purely self-supervised. Investigating how these static groups form—whether by color, geometry, or semantics—would be an intriguing direction for future work.

Dear Reviewer Wu7P,

Thank you for your thoughtful and detailed feedback on our submission. Your insights have been invaluable in refining our work, and we have addressed your comments comprehensively in our rebuttal.

With the discussion deadline approaching, we want to ensure that all your concerns are fully addressed. Please do not hesitate to reach out if you require further clarifications or analyses.

We understand you had concerns regarding some ambiguities in the initial submission. To address these, we have:

• Clarified how images $\hat{I}$ are computed and improved the design explanations of the motion keys and queries, with details provided in the updated manuscript.
• Conducted comprehensive ablation studies on key components, including the effect of the number of motion bases.
• Provided additional details to complement our initial description of baseline implementation details.
• Made 3D visualizations available on our project page to better illustrate temporal consistency and geometric accuracy.
• Clarified how point-tracking is achieved and explained the editing process in latent-STORM.

We hope these updates and clarifications have addressed your concerns and strengthened our submission.

If there are any further clarifications or analyses you would like us to provide, please let us know.

Additionally, we hope you could consider raising your score based on our rebuttal, as we believe these revisions, guided by your thorough review, have greatly improved the paper.

Thank you again for your thoughtful feedback and for helping us improve this work!

Best regards,

Authors

Dear Reviewer Wu7P03,

Thank you once again for your thoughtful feedback on our paper and the time you’ve dedicated to reviewing it.

We noticed that there is only one day left in the discussion period, and we would greatly appreciate any additional feedback you might have. If there are clarifications, further analyses, or experiments you’d find valuable, please let us know. We are eager to address any remaining concerns or suggestions to strengthen our work further.

We truly value the effort and insight you’ve already contributed and look forward to your follow-up response, including any reflection on the current score.

Thank you again for your time and support in this process. Please don’t hesitate to reach out with any further questions or thoughts.

Best regards,

The Authors