Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation

We sincerely thank Reviewer mAdS for the thoughtful and constructive feedback. We will incorporate the reviewer’s suggestions to further strengthen the paper in the final version.

More ablation studies

Training data sensitivity analysis

We analyzed the training data sensitivity for both the multi-view video generation and feedforward reconstruction modules below:

• Multi-view video generation analysis: The table below shows our ablation study on different training set combinations, tested on the object-centric Objaverse dataset and the dynamic scene Nvidia dataset. Each model was trained with identical settings (batch size of 128, 2k iterations). Our observations are as follows:
  • Data efficiency: Our method is data-efficient. For instance, a model trained solely on Objaverse generalizes reasonably well to the out-of-distribution Nvidia dataset.
  • Importance of 2D transformed video datasets: This is a crucial source for dynamic scene generation, yielding the second-best results on Nvidia when used as the sole training data.
  • Combined datasets: Training with all datasets combined delivers strong results across both scenarios.

• Feedforward reconstruction analysis:

In the table below, our ablation study evaluates how training data affects the feedforward reconstruction model. The original model is trained on five datasets: RealEstate10K, DL3DV, MVImageNet, ACID, and static cuts of Kubric. The accompanying table shows performance when the model is trained on each dataset individually. For Neural3DVideo, we fine-tune a dynamic variant derived from each baseline. Training on all datasets results in a noticeable improvement. With more computing power, one could possibly search over all possible subsets of these datasets to find the best possible results.

Limited ablations on stress cases and failure modes

We already discuss the limitations in the conclusion section (L326). We agree that it is a good idea to accompany the limitation discussion with qualitative results. Due to the text-based nature of this rebuttal, we are unfortunately unable to provide additional visual examples at this time. However, we commit to including a thorough discussion on failure and stress cases in the final version.

Marginal gain over 4Real-Video w/o 2D transformed videos?

We kindly encourage a re-examination of Table 3. Both our method and 4Real-Video utilize 2D transformed videos during training. Table 3 presents an ablation of our model without 2D transformed videos, yet it still surpasses 4Real-Video in visual quality. Our complete method, trained with 2D transformed videos, demonstrates a substantial improvement in cross-view consistency, while maintaining visual quality comparable to our model without 2D transformed videos.

We thank Reviewer f2R9 for the detailed and encouraging feedback. We will incorporate the reviewer’s suggestions in the final version of the paper.

Why sequential architectures performed poorly?

We have traced the issue to instability when the sequential model is trained on our mixed dataset. Its loss curve fluctuates more and converges to a higher value, reflecting less reliable/stable learning. Although the model performs reasonably on real scenes (Table 2 in the paper), it often misidentifies the white background that appears in about one-third of the training sequences, producing visible artifacts in the results. These background artifacts are simply the most obvious out of multiple failures; a closer inspection also reveals subtler issues such as blurred edges and temporal flickering. The sequential design exhibits an effective capacity that is lower than that of the fused model, making it harder to reconcile the differing background statistics. We will include these findings, along with loss curves and additional qualitative examples, in the revision.

Quantitative evaluation of sparse attention mechanism efficiency

We measured the runtime and peak memory consumption of various architectures. Our profiling was conducted on an A100 GPU using float32 precision, evaluating a single forward pass with 8 views and 61 frames at 288x512 resolution (resulting in 8x16x36x64 tokens after tokenization and patchification). Our findings indicate the following:

• A sequential architecture, which adds cross-view attention layers (similar to SynCamMaster), would increase network parameters by 50%. This leads to out-of-memory errors, given the already substantial 11B base video model.
• Another sequential model variant, which utilizes existing DIT blocks from the base video model but interleaves cross-view and cross-time attention dimensions, achieves faster runtime but sacrifices quality (see Table 2).
• A parallel architecture (similar to 4Real-Video) results in higher peak memory consumption and increased runtime.
• The proposed fused View-Time attention, when applied without block masking, leads to significantly slower runtime compared to its application with block masking.

Training time and rescources

• Training details for feedforward reconstruction: We used the publicly available weights and implemented the training code ourselves based on the released model of VGGT. The training is done on 32 A100 GPUs for around a day. We will provide a thorough explanation of the training and costs in the final version.

• Unclear details about the Kubric-based dataset: We followed the literature in rendering 32 views for each of the 3k scenes at different timesteps [1]. The views are sampled at the same altitude with constant azimuth angle increments. We’ll clarify and provide details in the final draft.

• Training details for multi-view video generation: Training cost for the video diffusion model was reported in the supplementary (see Ln.754), it was trained on 48 A100 GPUs for around 2 days, totalling around 96 GPU-days. We also experimented with a reduced training budget, using 32 A100 GPUs for about 20 hours (2k iterations). This yielded qualitatively competitive results, suggesting the method's reproducibility with fewer resources than initially reported. As reported above, our feedforward reconstruction model was trained on 32 A100 GPUs for around a day.

Cluttered layout

We will fix the formatting in the final version.

[1] Liang et al., “Feed‑Forward Bullet‑Time Reconstruction of Dynamic Scenes from Monocular Videos,” 2025

We appreciate Reviewer w5xW’s positive evaluation and constructive feedback. Our responses to each comment are detailed below, and the paper draft will be updated accordingly.

Unclear Dynamic Modeling in Feedforward Reconstruction

Our feedforward reconstruction model generates pixel-aligned Gaussian particles for each input frame from the multi-view video produced in the first stage, enabling novel view synthesis over time. Specifically, given an input multi-view video of resolution HxW with V views and T timesteps, the reconstruction model outputs V x T x H x W Gaussian splats. This is similar to other feedforward approaches for dynamic scenes—such as L4GM—and does not explicitly model motion, instead generating independent sets of Gaussian particles for each timestep. While we acknowledge that explicit motion modeling remains an important research direction, in this work we model motion implicitly through temporal attention layers. Notably, our framework is agnostic to the reconstruction backbone, allowing it to incorporate more advanced feedforward models as they emerge.

Evaluation Disconnect and Scalability Concerns

• FFD evaluated separately: Our method has two stages. We rigorously evaluate each component, achieving state‑of‑the‑art performance, and provide qualitative results for the combined stages in the paper as well as the supplementary material. Because of the 100 MB limit, we included fewer videos (without cherry‑picking) and will add more examples in the camera‑ready version. Below, we present a quantitative evaluation of the full pipeline on the Nvidia Dynamic Scene dataset, holding out views 1, 4, 7, and 10 at every timestep for evaluation. The optimization‑based baseline, 4DGS [1], is run on the first view at all timesteps plus the first timestep of every view (first row and first col of the 4D grid); in a second setting, we first complete the grid with the 4D video generator (while using the first column and the first row as input) and then optimize with 4DGS; in a third setting, we feed the generated grid to our feedforward reconstruction model, thereby applying the complete two‑stage pipeline. Our pipeline outperforms the baselines in quality and reduces reconstruction time from over an hour (4DGS) to under 3 seconds. Even if this or another optimization-based method could produce better visual results, it would not be an option due to their prohibitive runtime for every scene. To avoid potential issues with baseline depth/optical-flow priors failing on generated content, we utilized 4DGS as the standard prior-free method for dynamic scene reconstruction.

• FFD stage faces scalability issues (e.g., OOM at 16 views) without a mitigation strategy

In our experiments, our video generation part generates 8 to 15 views, depending on the setup. We can generate more views and timesteps in an autoregressive fashion. The feedforward model is perfectly capable of processing the original outputs of the video model. Both of the models can be applied autoregressively to support more views/timesteps. We will add thorough clarifications in the final version.

Training cost not reported

• Training cost for the video diffusion model was reported in the supplementary (see Ln.754), it was trained on 48 A100 GPUs for around 2 days, totalling around 96 GPU-days. We also experimented with a reduced training budget, using 32 A100 GPUs for about 20 hours (2k iterations). This yielded qualitatively competitive results, suggesting the method's reproducibility with fewer resources than initially reported.

• Training cost for feedforward reconstruction model: We construct our model based on VGGT, and fine-tune the model based on the released VGGT checkpoint. The training is done on 32 A100 GPUs for around a day. We will provide a thorough explanation of the training and costs in the final version.

• Compute-heavy design raises practicality concerns: We respectfully disagree with the concern that our training cost is excessive.

  • Our 4D video model was trained using 96 GPU-days, which is comparable to, or even lower than, several recent baselines. For instance, 4Real-Video required 32 A100s for 2 days (64 GPU-days), BTimer used 32 A100s for 4 days (128 GPU-days), GS-LRM needed 64 A100s for 3 days (192 GPU-days), LVSM used 64 A100s for 3 to 7 days (192–448 GPU-days), and LRM relied on 128 A100s for 3 days (384 GPU-days).
  • At inference, our framework generates a full grid of 288×512-resolution videos with 8 views and 29 frames using 40 diffusion steps in approximately four minutes, and performs feedforward reconstruction in just a few seconds, comparable to existing video generation methods. We also experimented with varying numbers of diffusion steps (see Table S2 in the Supplementary Material) and reported the corresponding runtimes. While standard diffusion-model distillation techniques could further reduce inference time, such optimization lies beyond the scope of the current work.

[1] Wu et al., “4D Gaussian Splatting for Real‑Time Dynamic Scene Rendering,” CVPR 2024.

We appreciate Reviewer 9auW's positive evaluation and constructive feedback. Our responses to each comment are detailed below, and the paper draft will be updated accordingly.

Efficiency comparison with baseline architectures

We measured the runtime and peak memory consumption of various architectures. Our profiling was conducted on an A100 GPU using float32 precision, evaluating a single forward pass with 8 views and 61 frames at 288x512 resolution (resulting in 8x16x36x64 tokens after tokenization and patchification). Our findings indicate the following:

• A sequential architecture, which adds cross-view attention layers (similar to SynCamMaster), would increase network parameters by 50%. This leads to out-of-memory errors, given the already substantial 11B base video model.
• Another sequential model variant, which utilizes existing DIT blocks from the base video model but interleaves cross-view and cross-time attention dimensions, achieves faster runtime but sacrifices quality (see Table 2).
• A parallel architecture (similar to 4Real-Video) results in higher peak memory consumption and increased runtime.
• The proposed fused View-Time attention, when applied without block masking, leads to significantly slower runtime compared to its application with block masking.

Clarification of masked self-attention

We aimed for mathematical precision in defining the binary mask in Eq. 1, but acknowledge that this resulted in a somewhat complex formulation. We will revise the presentation to improve clarity. Conceptually, however, the idea is straightforward: each token is indexed by (v,t,x,y), and tokens are allowed to attend to one another if they share the same view v, the same timestamp t, or both. Attention between token pairs that do not satisfy any of these conditions is masked out. For the reviewers' specific questions:

• How spatial tokens within the same (v,t) coordinate interact? They attend to each other, the same way as the unmodified 3D self-attention in the original base video model. This is the case of tokens sharing both the same view and same time stamp.
• Does mask support learnable attention weights, how gradient flow is affected? In an attention layer, the learnable attention weights refer to the MLP parameters used to compute the query (Q), key (K), and value (V) matrices. In the case of sparse attention, these same weights are updated during training. However, token pairs whose attention is masked out do not contribute to the gradient. Only token pairs for which attention is computed affect the gradient update. This follows the standard behavior of masked attention operations and is fully supported by the FlexAttention implementation in PyTorch.

Comparing to other 4D coordinate embedding

We implemented a new baseline that incorporates a learnable positional embedding for the view dimension. Specifically, for each token indexed by (v, t, x, y), we compute separate positional embeddings for the view v and the spatiotemporal coordinates (t, x, y). For the view dimension, we apply sinusoidal encoding with 16 learnable frequencies, followed by a two-layer MLP to produce the final view embedding. For (t, x, y), we adopt the 3D RoPE (Rotary Positional Embedding) used in the original base video model. To integrate both positional embeddings into the attention mechanism, we first add the view embedding to the key and query vectors, and then apply the 3D RoPE. This ensures that the RoPE is applied correctly while still incorporating view-specific information.

We compare this baseline against our proposed embedding under identical training and testing settings (see table below). We observe that the model with learnable view embeddings converges more slowly during training. While it achieves comparable performance on dynamic-scene datasets such as the NVIDIA dataset, it fails to produce correct backgrounds (e.g., white backgrounds in the Objaverse dataset) resulting in catastrophic failures. We hypothesize that this may be due to confusion arising from the mixed nature of the training data. Further investigation, such as tuning the training schedule or dataset composition, may help address this issue. In conclusion, the proposed embedding is empirically more stable to train, achieves competitive or superior quality, and introduces no additional parameters.

Citation to concurrent work (Nutworld)

We thank the author for recommending the Nutworld paper. We will include it as a related work for feedforward reconstruction.