Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos

[W1] Dependencies on pre-computed camera poses and requires significant computation for the initial training.

1. Most feed-forward reconstruction models need camera poses as input. Particularly in monocular dynamic scene reconstruction settings, without camera poses will add more ambiguity to the dynamics of the scene, so we use it in this work. And we would like to leave the model without camera poses for future work. We will also release the full method and code for performing camera annotations for monocular video soon.
2. The training cost for BTimer (≈ 256 GPU‑days) is on par with other state‑of‑the‑art feed‑forward 3 D reconstruction methods—e.g., LVSM (ICLR’25) (≈ 384 GPU‑days), LRM (ICLR’24) (≈ 384 GPU‑days), and GS‑LRM (ECCV’24) (≈ 192 GPU‑days). Like these works, our model operates as an amortised algorithm: once training is complete, inference is virtually negligible. When inference cost is taken into account, per‑scene optimization pipelines rapidly become more expensive, and the gap widens as the number of scenes grows (see Fig. S1 in the supplementary).

[W2] Implicit Motion

1. Modeling dynamics implicitly allows our model to capture flexible and complex dynamics, such as deformation with topological changes, and complex natural phenomena such as fire, waves, smoke, and dust. This is evident in our supplementary visualizations on SORA and DAVIS datasets.
2. Despite using implicit time encoding, our model can still recover decent 3D deformation and scene flows as evidenced in Figure S5 and S6 of the supplementary, where the dynamic structure of the scenes is clearly recovered. This indicates that the model learns meaningful motion representations from time embeddings, even without explicit motion supervision.
3. Our approach aligns with several prior works that successfully use implicit time modeling for dynamic scenes, including ST-NeRF (CVPR’21), DyNeRF (CVPR’22), and HexPlane (CVPR’23).

[Q1] The pipeline for annotating internet videos—using SAM to mask dynamic objects before applying DROID-SLAM—is a very clever approach to leverage a powerful static-scene SLAM system. However, this relies on the segmentation quality and treats motion as an outlier to be removed, rather than data to be modeled.

Thanks for the thoughtful comment. We generally observe that the segmentation quality from SAM is sufficient for the majority of internet videos. In challenging cases where semantic segmentation may fail, the original outlier rejection mechanism in DROID‑SLAM provides a complementary safeguard by regressing per‑pixel weights, which we use as a supplement to down‑weight and prevent them from corrupting the pose estimation.

It is important to note that, in this pipeline, our primary objective is to obtain reliable camera poses. As such, explicit motion modeling at this stage is not strictly required, which is why we remove them rather than model them directly. Motion information is preserved in the input frames and later learned by the BTimer network during reconstruction.

[Q2] I'm curious if the authors experimented with SLAM systems that explicitly model dynamic objects? For example, methods that track moving objects separately or represent dynamic parts of the scene with their own motion models. What was the rationale for choosing the current pipeline over these alternatives? Was it primarily for robustness and scalability on diverse internet videos, or did dynamic-aware SLAM methods prove less reliable in practice?

Thank you for the question. We did consider SLAM‑style pipelines that treat dynamic objects separately, but found them ill‑suited to our target setting of unconstrained, monocular in-the-wild videos:

1. Limited robustness and scalability. Dynamic‑aware SLAM pipelines require reliable depth, motion segmentation, and object tracking. In practice these steps are not robust enough (PGDVS) and fail on texture‑poor, fast‑moving or out-of-distribution clips. Per‑object tracking and optimisation make such systems slower than a single feed‑forward pass and difficult to scale to large datasets.
2. And SLAM methods typically assume known and accurate camera intrinsics, which are unavailable for the vast majority of in‑the‑wild videos. With the goal of pursuing robustness, scalability and a complete generalizable framework, we make this framework design.

[Limitation1] Gaps in Ablation and Analysis

Thank you for the suggestion. Although we have not yet performed a dedicated robustness study, several design choices (please refer to answer of Question1) already mitigate most SAM + SLAM annotation errors. We will add a detailed discussion of these factors in the final version.

[W1] Benchmark Comparison

Thank you so much for this suggestion.

1. Baselines selection: we kindly clarify that we are not cherry-picking baselines. For each benchmark, we compare with methods that have official results on that benchmark, to ensure the comparison is rigorous and fair. This is exactly the same convention followed by previous dynamic‑scene reconstruction works (e.g., MoSca, CAT4D, PGDVS, Casual‑FVS, RoDynRF), where they likewise report different baseline sets across DyCheck and NVIDIA baselines for the same reason. If additional official results appear, we are happy to include these additional comparisons in the revised version.

2. Metric Comparison: For the same reason as baselines selections, to enable direct, fair comparisons with the published results of prior work, we follow each benchmark’s established protocol and adopt exactly the metrics that the existing baselines report. On NVIDIA Dynamic Scenes, prior works (MoSca, Casual‑FVS, RoDynRF, MonoNeRF, DynNeRF) routinely list PSNR, LPIPS but not SSIM. We are happy to provide our SSIM score (0.835), however we didn’t find any official reported SSIM score from our compared baselines. FPS is reported in Casual‑FVS, so we adopt the same set. The reconstruction time for both two benchmarks have been reported in PGDVS.

   In any case, we will include complete metric comparison for those that are officially available in our revised version. As for now, we additionally evaluate our methods all three different evaluation protocols used by prior works on Dynamic Scenes, so that Reviewers can conveniently compare.

   	Reference	PSNR↑	SSIM↑	LPIPS↓
   Protocol A*	DynNeRF [1]	25.82	0.835	0.086
   Protocol B	NSFF [2]	26.40	0.856	0.071
   Protocol C	DynIBaR [3]	26.31	0.852	0.073

   *We used this evaluation protocol in the main paper.

3. Considering that the focus of our work is on feed-forward reconstruction setting, we believe the reported benchmarks are sufficient to demonstrate that BTimer matches or surpasses existing per-scene optimization methods in quality while achieving several‑orders‑of‑magnitude faster runtime. Still, thank you so much for the suggestion.

[1] Dynamic View Synthesis from Dynamic Monocular Video, ICCV 2021

[2] Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes, CVPR 2021

[3] DynIBaR: Neural Dynamic Image-Based Rendering, CVPR 2023

[W2] Runtime performance

Thank you for this valuable suggestion. To provide a more rigorous assessment of runtime, we have conducted an additional study varying both the number of context frames and the input resolution.

The new results show that BTimer consistently remains in the second range (even at 512 × 512 resolution with 12 context frames), and is still several orders of magnitude faster than per‑scene optimisation pipelines. We will include these studies in the revised manuscript.

[W3] pixel-aligned prediction vs. XYZ prediction

Thank you for raising this point. Canonical XYZ prediction is commonly used in object‑centric cases for being bounded (LGM, L4GM), however pixel-aligned prediction is popular in large unbounded scene for being easy to optimize (pixelSplat, GS-LRM). Applying canonical XYZ prediction to scene data is a valuable direction that we would like to explore in future work. And we will include the comparison discussion between them in our revised version.

[Q1] For a given in-the-wild video, what is the prescribed method for estimating camera poses and intrinsics?

Given an in-the-wild video, we first apply per-frame Segment Anything Model to mask out pixel regions corresponding to the dynamic objects. The SAM model is prompted with several predefined categories of movable objects. In the meantime, we initialize the camera intrinsics using the GeoCalib model. We then apply DROID-SLAM on top of the video to estimate the camera poses, and the camera intrinsics is refined along with the poses in the global Bundle Adjustment process. We will release the full method and code for performing such camera annotations soon.

[Q2] Do the authors intend to release the source code and model checkpoints for re-implementation?

Yes, the code and pretrained model will be released once accepted.

[W1,Q1] What is the contribution of the method beyond stitching existing components together?

We would like to clarify several points regarding our contributions:

1. We propose the first feed‑forward dynamics scene reconstruction model with bullet‑time formulation. Prior feed‑forward methods have been confined to static scenes due to (a) the challenge of modeling dynamics in a feed-forward manner and (b) the lack of large‑scale 4D supervision data.

   • Our bullet‑time formulation extends well beyond a “standard ViT + time embedding” since it addresses both of the above issues in a novel manner, by training the network to aggregate context frames and predict the scene at an arbitrary target timestamp. This implicitly encourages the model to become motion-aware and capture scene dynamics. To our knowledge, our formulation is the first feed-forward dynamic reconstruction method.
   • More importantly, this formulation naturally unifies the static and dynamic reconstruction scenarios that allows us to pre-train our model on large amounts of static scene data and scales effectively across datasets, thus no longer being constrained by scarcity of 4D data. This scalability is key to generalization and performance, as shown in our results across static and dynamic benchmarks.

   We believe this formulation is both conceptually novel and practically impactful, offering a significant step towards feed-forward real-time dynamic scene reconstruction.

2. Although our design is inspired by LVSM, the NTE module differs in purpose and formulation:

   • NTE consumes the context frame, time embeddings for both context and target timestamps, and relative pose, whereas LVSM operates without time embedding.
   • LVSM is designed for static reconstruction, while NTE is used within a dynamic‑reconstruction pipeline to predict missing intermediate frames under fast or sparse motion.
   • NTE functions as an image‑space motion interpolator that complements the bullet‑timer, refining its 3DGS predictions and improving temporal consistency in challenging dynamic scenarios.

   These differences make NTE a novel, task‑specific component rather than a direct reuse of LVSM.

3. Our three‑stage curriculum is deliberately crafted to leverage abundant static data while gradually introducing motion complexity. We also introduce interpolation losses which ensures a stable convergence of the model.

4. BTimer unites efficiency, robustness and fidelity. It delivers orders of‑magnitude faster inference than per‑scene optimization pipelines while preserving comparable quality. A single model generalises across a broad range of datasets with substantial distribution shifts, including real-world dynamic scenes, static scenes, AI-generated content (e.g., from Sora), and synthetic 4D environments (e.g., Kubric4D), whereas prior works train separate models for each domain.

Together, we believe our contributions form a cohesive and non-trivial advancement in feed-forward dynamic scene reconstruction and address long-standing challenges in scalability, supervision, and real-time inference.

[W2] Mixed qualitative performance

We appreciate the reviewer’s careful inspection of the qualitative videos and would like to clarify the observed artifacts:

1. Our model interpolates views by aggregating information across the context frames; consequently, its accuracy is proportional to the effective baseline and pose fidelity in the input. Most Sora clips contain only subtle camera motion and lack accurate pose estimates, making dynamic‑content extrapolation shown in bullet-time setup intrinsically ill‑posed. In scenes where the baseline is larger and the poses are more reliable (e.g., amalfi‑coast, cloud‑man, robot), the reconstructions are visibly sharper. We will highlight this dependency and add analysis in the revision.
2. Dycheck is a challenging dataset that features generic, high diverse dynamic scenes(MoSca [CVPR 2025]), where even many optimization‑based methods struggle. The iphone 7 sequence, in particular, shows a hand moving much faster at close range, this is an edge case for many reconstruction techniques. Despite this, our feed‑forward model,
   • outperforms several per‑scene optimisation approaches on this very clip (see the DyCheck evaluation page https://xiaoming-zhao.github.io/projects/pgdvs/videos/dycheck_eval/apple/index.html, where those approaches exhibit even more severe artefacts),
   • matches or exceeds their quantitative scores across the entire DyCheck set,
   • performs well on other hand sequences (iphone4, 6, 8, 33) with high fidelity. We will add a brief discussion of this failure case and include comparisons to existing methods in revision.
3. As noted in Point 4 of our response to W1/Q1, BTimer shows strong robustness, efficiency, and reconstruction quality across diverse datasets.

[W3.1, Q3] Comparison with DynIBaR.

1. DynIBaR achieves high quality by performing hours of per‑scene optimisation (≈ 300 h per NSFF sequence) and by utilising additional supervision signals (optical flow + depth). In contrast, BTimer is a fully feed‑forward system: it reconstructs a scene within 1 second and is trained using only photometric loss. Given the vastly different computational budgets and data requirements, we belive that a head‑to‑head comparison is not strictly necessary for establishing our contribution.

2.  Nevertheless, we evaluate our method using DynIBaR’s protocol and compare it with the official DynIBaR results and the reproduction by PGDVS [ICLR 2024] (*) for reference.

   	PSNR↑	SSIM↑	LPIPS↓
   DynIBaR	30.86	0.957	0.027
   DynIBaR*	29.35	0.934	0.0621
   BTimer	26.31	0.852	0.0730

[W3.2, Q2] Not all baselines are evaluated on both of the used datasets.

Thanks for this suggestion.

1. Baselines selection: we would like to clarify that we are not cherry-picking baselines. For each benchmark, we compare with methods that have official results on that benchmark, to ensure the comparison is rigorous and fair. This is exactly the same convention followed by previous dynamic‑scene reconstruction works (e.g., MoSca, CAT4D, PGDVS, Casual‑FVS, RoDynRF), where they likewise report different baseline sets across DyCheck and NVIDIA baselines for the same reason. If additional official results appear, we will incorporate them in the revision. Due to the limited rebuttal time, we were not able to re‑run T‑NeRF on the NVIDIA dataset (or DNeRF on DyCheck). However, we are happy to include these additional comparisons in the revised version.
2. Metric Comparison: For the same reason as baselines selections, to enable direct, fair comparisons with the published results of prior work, we follow each benchmark’s established protocol and adopt exactly the metrics that the existing baselines report. On NVIDIA Dynamic Scenes, prior works (MoSca, Casual‑FVS, RoDynRF, MonoNeRF, DynNeRF) only list PSNR, LPIPS but not SSIM. We are happy to provide our SSIM score (0.835), however we didn’t find any official SSIM score from our compared baselines. FPS is reported in Casual‑FVS, so we adopt the same set. The reconstruction time for both two benchmarks have been reported in PGDVS. Anyhow, we will include complete metric comparison for those that are officially available in our revised version.
3. Considering that the focus of our work is on feed-forward reconstruction setting, we believe the reported benchmarks are already sufficient to demonstrate that BTimer matches or surpasses existing per-scene optimization methods in quality while achieving several‑orders‑of‑magnitude faster runtime.

[Q4] meaning of L164.

Since the 3D position of each Gaussian is obtained through pixel-aligned unprojection (L121-122), this introduces a geometric bias that every Gaussian is anchored to a corresponding viewing ray of the input context frame. When we query an intermediate timestamp that has no corresponding context frame, the predictor struggles to predict the gaussian at that timestamp. In supplementary (L84) we train a variant of BTimer that removes this bias, which demonstrates that our BTimer formulation can successfully recover the 3D deformation and scene flow (Fig. S5-6 of supplementary).

[Q5] The paper uses a low resolution of 480 × 270 for for their quantitative evaluation and claims 150 FPS for NVIDIA Dynamics Scene (Tab. 1). How does this number change for high quality rendering (eg. Full-HD or 4K)?

We additionally evaluate at 912x512 on Dynamic Scenes using the NSFF protocol. (Previous work evaluate at no higher than 288x540.) Due to limited rebuttal time, we test the existing 512x512 model without retraining a high resolution one, so quality drops somewhat. Inference rises from 0.44s to 4.48s due to the larger token count, yet it remains orders of magnitude faster than per-scene optimisation.

[Q6] BTimer and 4D-GS rely on the same underlying representation. Can you give an explanation why BTimer is nearly 3x faster at rendering?

There are mainly two reasons:

1. 4D‑GS stores degree‑3 spherical‑harmonic (SH) coefficients for every Gaussian, while BTimer removes this overhead by storing a single RGB value, which cuts the computation and memory bandwidth per splat.
2. 4D-GS did not effectively reconstruct the scene from monocular video, hence more particles are used in order to overfit each view, causing the speed performance drop. Please note that we use their default configuration and directly trained on the NVIDIA benchmark, and both are tested on a single A100 GPU.

I thank the authors for their extensive rebuttal. The following questions have been sufficiently addressed: W2, W3.2/Q2, Q4, Q5, Q6.

• [W1/Q1]: The rebuttal makes a strong case that the method has conceptual novelty, but the model still heavily relies on existing architectures and therefore has a limited technical novelty. Anyhow, I agree that there is a contribution in the integration and evaluation of the components.
• [W3.1/Q2]: I am aware of the massive computationally effort it takes to utilize DynIBaR but I think it is importance to showcase the maximum possible performance for a posed problem to put the own performance into perspective. Therefore, I think it is important to include the results of DynIBaR since they are the actual SotA without additional restrictions such as limited computational budget/requirement of fast per-scene deployment. This does not mean that BTimer can claim SotA for feedforward reconstruction or a limited computational budget for inference.

[W1.1] Incremental Contribution of bullet-time embedding and NTE module

We respectfully disagree with the assessment that our contributions are incremental.

1. Prior feed-forward models are limited to static scene reconstruction mainly due to two key challenges: (a). the complexity of modeling scene dynamics in a single forward pass, and (b). the scarcity of large-scale 4D dynamic supervision. Our proposed bullet-time formulation directly addresses both issues in a novel and elegant manner.

   • the bullet-time design allows our model to predict the scene at an arbitrary bullet time by aggregating information from context frames. This training objective implicitly encourages the model to become motion-aware and capture scene dynamics. This is the first feed-forward dynamic reconstruction formulation that has never been explored before.

   • More importantly, this formulation naturally unifies the static and dynamic reconstruction scenarios that allows us to pre-train our model on large amounts of static scene data and scales effectively across datasets. This scalability is key to generalization and performance, as shown in our results across static and dynamic benchmarks.

   We believe this formulation is both conceptually novel and practically impactful, offering a significant step towards feed-forward real-time dynamic scene reconstruction.

2. We propose the NTE module to further enhance reconstruction quality, particularly under complex and fast motions. It introduces a novel design by conditioning on the context frame, interpolated timestamp embedding, and relative pose. This design complements the bullet-time prediction and can be seamlessly integrated into our framework.

3. Moreover, we also introduce curriculum training at scale and interpolation supervision, which are critical to ensuring stable convergence. As demonstrated in our ablation studies, each of these components contributes meaningfully to the overall performance of the model.

Taken together, we believe our contributions form a cohesive and non-trivial advancement in feed-forward dynamic scene reconstruction and address long-standing limitations around scalability, supervision, and real-time inference.

[W1.2] Reliance on large-scale static and dynamic datasets raises concerns about generalizability.

1. To rigorously evaluate the generalization ability of our model, we conduct extensive experiments across a broad range of datasets, including real-world dynamic scenes, static scenes, AI-generated content (e.g., from Sora), and synthetic 4D environments (e.g., Kubric4D). These datasets exhibit substantial domain gaps, yet our model maintains consistent and strong performance, achieving results competitive with per-scene optimization methods and significantly outperforming previous feed-forward baselines. We believe this provides compelling empirical evidence of robustness and cross-domain generalization.

2. To be the same with most recent feed-forward reconstruction methods (e.g., GS-LRM [1], LGM [2], DUSt3R [3], VGGT [4]), our model leverages large-scale datasets to enhance generalization capacity. This reliance is not a limitation, but a necessary component for real-time, generalizable scene understanding at scale. We emphasize that our design further amplifies the benefits of such pretraining by unifying static and dynamic reconstruction within a single formulation.

3. We further include additional experiments to demonstrate transfer performance across datasets. We simplify the training setting and only use RE10K in the static stage and a mix of Kubric4D and RE10K in the dynamic stage. The model is evaluated on the Dynamic Scenes dataset. We highlight that Kubric4D is a synthetic dataset with rigid motion, while Dynamic Scene is a real-world dataset with articulated and deformable motion. Despite the significant domain difference, training on Kubric4D substantially improves all metrics on Dynamic Scenes, which shows that the dynamic representation learned can be effectively transferred across very different datasets.

   	PSNR↑	SSIM↑	LPIPS↓
   RE10K	21.62	0.724	0.257
   RE10K + Kubric4D	23.61	0.755	0.214

[1] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting, ECCV 2024

[2] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, ECCV 2024

[3] DUSt3R: Geometric 3D Vision Made Easy, CVPR 2025

[4] VGGT: Visual Geometry Grounded Transformer, CVPR 2025

[W2] Weak motivation of timestamp embeddings over explicit motion modeling

Timestamp embeddings offer important advantages over explicit motion modeling.

1. It allows our model to capture flexible and complex dynamics, such as deformation with topological changes, and complex natural phenomena such as fire, waves, smoke, and dust. This is evident in our supplementary visualizations on SORA and DAVIS datasets.

2. Despite using implicit time conditioning, our model can still recover decent 3D deformation and scene flows as evidenced in Figure S5 and S6 of the supplementary, where the dynamic structure of the scenes is clearly recovered. This indicates that the model learns meaningful motion representations from time embeddings, even without explicit motion supervision.

3. Our approach aligns with several prior works that successfully use implicit time modeling for dynamic scenes, including ST-NeRF [5], DyNeRF [6], and HexPlane [7].

[5] Space-Time Neural Irradiance Fields for Free-Viewpoint Video, CVPR 2021

[6] DyNeRF: Neural 3D Video Synthesis from Multi-view Video, CVPR 2022

[7] HexPlane: A Fast Representation for Dynamic Scenes, CVPR 2023

[Q1] Compared with CAT4D and MoSca

1. Our primary focus is on fully feed-forward dynamic scene reconstruction, while both CAT4D and MoSca rely on lengthy per-scene optimization to achieve 4D reconstruction. CAT4D’s code and pretrained weights are not available, which precludes a broader empirical comparison. And, in MoSca’s case, it requires additional 2‑D tracks and depth supervision for optimization. Hence, a direct comparison is not strictly required.

2. Notwithstanding the differing problem settings and supervisions, we make a comparison on the NSFF benchmark, the only common dataset on which CAT4D and MoSca report results. Using their own reported scores, our method significantly surpasses CAT4D and is on par with MoSca, while our model is trained with only photometric loss, and runs in a feed-forward manner.

   	PSNR↑	SSIM↑	LPIPS↓
   MoSca	26.72	-	0.070
   CAT4D	21.97	0.683	0.121
   BTimer	25.82	0.835	0.086

[Q2.1] Quantitative effect of NTE module

1. NTE is designed to improve reconstructions when the temporal frame is sparse or motion is fast, by refining predictions at intermediate timestamps. On standard dynamic benchmarks such as Dynamic Scenes, frames are dense and motion magnitude is limited; the standard protocal also does not evaluate in-between timestamps.

2. We have shown qualitative ablation in Fig6. To provide quantitative evaluation, we additionally follow the interpolation evaluation protocol in NSFF by extracting every other frame from the original Dynamic Scenes dataset. w/ GT means using the ground truth in-between frames which shows the upper bound of the interpolation model. NTE effectively improves on PSNR and SSIM. Note that the slight drop on LPIPS is expected, since perceptual metrics will worsen with interpolation [8].

   	PSNR↑	SSIM↑	LPIPS↓
   w/o NTE	25.73	0.844	0.0786
   w/ NTE	25.97 (+0.24)	0.846 (+0.02)	0.0791 (+5e-4)
   w/ GT	26.29 (+0.56)	0.851 (+0.07)	0.0734 (-5.2e-3)

   [8] The Perception-Distortion Tradeoff, CVPR 2018

[Q2.2] Reason for outperforming direct 3DGS prediction in fast-motion scenarios.

As noted in L160–167, when temporal frames are sparse or motion is fast, the frame corresponding to an intermediate timestamp is absent from the context. Due to the inductive bias of pixel‑aligned 3D‑Gaussian predictor, the model struggles to produce smooth transitions. Since the NTE module is not subject to this 3D bias, it synthesizes the missing intermediate view accurately; we then pass this NTE output to BTimer for the final reconstruction.

[Q3] The model uses 12 context frames. How does BTimer scale to longer videos with more frames or higher temporal variability?

BTimer can accommodate lengthy or highly dynamic sequences through several practical strategies:

1. At test time we apply BTimer in a sliding window manner; thus, memory usage and latency remain constant, regardless of the total video length.
2. We can subsample representative key frames and pair them with the context frames corresponding to the target timestamp as input to the model. This reduces computational load while preserving global coherence.
3. The architecture readily accepts more frames. By replacing the standard ViT blocks with recent Mamba‑2 blocks, as adopted in Long‑LRM (ICCV 2025), we can process substantially more tokens efficiently.

Hence, these strategies allow BTimer to scale gracefully to longer videos and higher temporal variability without requiring fundamental architectural changes.