We thank the reviewer for their insightful feedback and recognition of our work. We appreciate their noting that "the method is simple, intuitive, and elegant", "the evaluation show strong quantitative results" and "the ablation studies validate the effectiveness of design choices".

 

Issues of clarity We deeply appreciate your constructive and detailed feedback. We will fix these issues in a revised version of the paper.

• The dual view sampling: As mentioned from line 182 to 185, we sample $Q=8$ output images at $\frac{Q}{2} = 4$ timesteps for ground-truth supervision. Specifically, at each timestep, we sample 2 images from 2 camera trajectories. This dual view supervision is essentially a multi-view (two-view) supervision that enhances both geometry and novel view synthesis quality. In the revised version of the paper, we will strictly differentiate between the terms "timestep" and "frame."
• View selection: For videos of static scenes, frames sampled at different timesteps provide sufficient multi-view supervision for novel view synthesis. However, for videos of dynamic scenes, simply using frames sampled at different timesteps can cause ambiguity, as both cameras and objects are moving. Therefore, we employ dual view supervision to sample 2 camera poses at the same timestep for rendering supervision.
• Role of reference views: The role of reference views is to increase the input baseline, which is important for both geometry and view synthesis quality. We may achieve sufficiently large baselines if the camera moves enough across a long temporal span. However, naively increasing the input video length would add too many input frames, leading to prohibitively large GPU memory consumption. Therefore, we sample $K=4$ extra temporally distant reference frames to help increase the baseline. A similar strategy was used in BTimer. We will further acknowledge this prior work in the revised version of the paper.
• Flow chaining procedure We agree that a visualization would be helpful, but we did not include it due to the page limit. We will add a better visualization in the revised version of the supplementary material. Our predicted per-Gaussian deformation field can deform Gaussians to every input timestep, not just the temporally downsampled key timesteps, and therefore will not cause issues in flow chaining.
• Union instead of sum: Thanks for the detailed comment! We will correct this in a revised version of the paper.
• Opacity and rotation deformation: For rotation, the per-pixel deformable 3D Gaussians are densely distributed in the foreground area. Each Gaussian typically has a small covariance. Therefore, we empirically did not observe any impact from fixed rotation. For opacity, our pilot study indicates that allowing changes in opacity enables the model to "fake" dynamic appearances without accurately predicting the deformation field, which negatively impacts tracking accuracy. We will add this analysis to the revised version of our paper.

 

Comparisons with PGDVS While DGS-LRM performs better in some examples with complex motion, such as the pinwheel in Figure 4, we agree that PGDVS generally recovers more detailed textures, as indicated by the quantitative data. In the revised version of the paper, we will tone down the use of the word "comparable" and be more specific about the properties of each method. We would also like to emphasize that DGS-LRM is three orders of magnitude faster at inference compared to PGDVS, which requires several hours for inference.

 

First feed-forward method We emphasize that DGS-LRM is the first feed-forward method that predicts deformable 3D Gaussians, supporting both novel view synthesis and spatial tracking simultaneously. Neither BTimer nor L4GM predicts deformation fields. Regarding BTimer, we fully agree that it would be an informative comparison, but their code and trained model are not currently available. Based on NeurIPS policies, we will remove the word "concurrent" when describing BTimer.

 

Ablation studies

• For depth supervision, our pilot study shows that our dual view supervision already provides sufficient geometry constrain, and adding depth supervision does not significantly impact the reconstruction quality. We simply keep it as we have ground-truth depth data.
• For scene normalization, both methods normalize the scene using the structure-from-motion depth for a fair comparison. Such a depth is exclusively used in scene normalization to align the scene scale to the benchmark unit. We will include this implementation detail in the revised paper.

 

Qualitative figures While the baselines in Figure 5 are relatively small, we did present view synthesis results with larger baselines in Figure 4 and the supplementary material, where DGS-LRM demonstrates competitive reconstruction quality. For Figure 6, we agree that adding ground truth flow would be useful, and we will include it in the revised version of the paper. Thank you again for the constructive feedback!

We thank the reviewer for their constructive feedback and recognition of our work. We appreciate their noting that our method "tackles a very timely topic and a very challenging task", "generalizes reasonably well to the test data" and achieves "strong performance in 3D tracking".

 

Writing of the method section Thank you for the constructive feedback! We noted that reviewer qSmi mentioned that "our paper is mostly well-written and easy to follow" and provided several suggestions to improve the clarity of the paper. We will include changes in our rebuttal to their review in a revised version of the paper, including more detailed explanations of key concepts, such as dual view supervision and reference views, as well as a better visualization for the flow chaining method.

 

Model design Thank you for the insightful feedback! First, we would like to highlight that our method is one of the first attempts to build a feed-forward approach for monocular dynamic scene reconstruction. It is also the first method to jointly predict both 3D Gaussians and deformation fields, supporting novel view synthesis and tracking simultaneously.

• Scene flow: Our deformation prediction similarly follow the common learning-based 2D and 3D flow prediction methods that also heavily benefit from direct flow supervision, with representative works such as CoTracker [1], SpatialTracker [2], RAFT [3], TAPIR [4], as well as classical works like FlowNet [5] and FlowNet 2.0 [6]. In our pilot study, we put significant effort into testing whether the model could learn deformation fields from rendering loss supervision. Finally, we concluded that ground-truth supervision is necessary to ensure physical meaningful tracking in the monocular video setting and we incorporate them as default in this work. We believe future work can relax this constraints and push this direction further.
• Large number of 3D Gaussian points:While it is true that a per-pixel Gaussian is an over-complete representation, it offers unique advantages such as fast convergence and high-quality texture details. Consequently, numerous recent feed-forward reconstruction methods adopt it as the scene representation, including PixelSplat [7], GS-LRM [8], GRM [9], Long-LRM [10], FLARE [11], etc. DGS-LRM follows this trend. For a 1-second video with 4 key frames, DGS-LRM generates $512 \times 512 \times 4 = 10^{6}$ Gaussian points. We have empirically found that this does not cause memory issues under our current settings. To scale up to higher resolutions, we may adopt Long-LRM's architecture and strategy, using opacity values to filter out redundant Gaussians, which we leave for future work.
• Dynamic sparsity: We have empirically found that a per-pixel deformation field fits well with a per-pixel Gaussian representation. We attempted to further explore dynamic sparsity by predicting the motion field from a K-plane representation in our pilot study, but found that it converged very slowly.

 

Comparisons with PGDVS We agree PGDVS is a strong baseline but we share a different objective. Our DGS-LRM is a general feed-forward method that emphasize on learning priors from data. It can run three orders of magnitude faster at inference while being closely on par with quality delivered by PGDVS. We observe that DGS-LRM can perform slightly better in some cases involving complex motion, such as the pinwheel example in Figure 4. This improvement is due to our method predicting more accurate deformation fields. We also note that PGDVS reconstructs texture details more effectively, as reflected in both the qualitative and quantitative results, which serves as a strong reference for learning based method to catch up. In the revised version of the paper, we will avoid using the phrase "comparable quality" and instead provide a more specific description of the advantages of each method.

We will add the following descript of PGDVS to the related works:

PGDVS achieves high-quality novel view synthesis on the DyCheck benchmark while significantly reducing reconstruction time compared to many prior optimization-based methods. It leverages off-the-shelf depth and optical flow estimators for initialization. Uniquely, it renders novel views at different timestamps via image-space warping and aggregates results using dynamic object masks, which produce sharp appearances. However, the method still requires hours to build its representation, and in some challenging cases with complex motion, warping and masking errors can cause temporal flickering artifacts. In our work, we focus on an end-to-end feedforward representation that can deliver similar outputs while being orders of magnitude faster in inference. This provides the potential to run at scale.

 

Handling invisible region Since DGS-LRM model is supervised with ground-truth scene flow, it has the ability to recover parts of dynamic objects that are partially observation in a subset of frames. For regions that has never been seen, we require generative prior to hallucinate its appearance, which is beyond the scope of this paper.

 

Scalability Thanks for the insightful feedback! We would like to re-emphasize DGS-LRM is one of the first attempts tackling this challenging problem and it shows competitive generalization ability as well as tracking and view synthesis accuracy on real-world videos under our current setting.

• Scaling to high-resolution While our current model achieves competitive view synthesis quality without causing memory issues, further scaling up resolution is possible with better training strategies and novel network architectures. Long-LRM manages to decode 32 images with resolution $960 \times 540$. Similar strategy can be adopted by DGS-LRM and we leave this for future works.
• Non-rigid motion Although our current simulation pipeline only contains rigid motion, the generated per-pixel deformation fields are actually diverse and comprehensive. Qualitative experiments also demonstrate our model's generalization ability on non-rigid motion, such as in Teddy and Apple examples. It is also possible to utilize our blender pipeline to simulate non-rigid deformable objects, which we leave for future works.

 

Response to questions:

Q1. Bluriness: As shown in the Supplementary, PGDVS has visible flickering deformation artifacts. Our learning based framework may obtain better deformation but fall short in the textural sharpness at this moment. It is worth noting our inference times a few orders of magnitude faster due to the nature of feedforward prediction, which is not reflected in the video comparisons.
Q2. Union instead of sum: Thanks for detailed suggestion, we will correct it in a revised version.

 

[1] Karaev, Nikita, et al. "Cotracker: It is better to track together." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.

[2] Xiao, Yuxi, et al. "Spatialtracker: Tracking any 2d pixels in 3d space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Teed, Zachary, and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow." European conference on computer vision. Cham: Springer International Publishing, 2020.

[4] Doersch, Carl, et al. "Tapir: Tracking any point with per-frame initialization and temporal refinement." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Dosovitskiy, Alexey, et al. "Flownet: Learning optical flow with convolutional networks." Proceedings of the IEEE international conference on computer vision. 2015.

[6] Ilg, Eddy, et al. "Flownet 2.0: Evolution of optical flow estimation with deep networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[7] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[8] Zhang, Kai, et al. "Gs-lrm: Large reconstruction model for 3d gaussian splatting." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[9] Xu, Yinghao, et al. "Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[10] Ziwen, Chen, et al. "Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats." arXiv preprint arXiv:2410.12781 (2024).

[11] Zhang, Shangzhan, et al. "Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We thank the reviewer for their insightful feedback and recognition of our work. We appreciate their noting that "the proposed method across both NVS and Point tracking tasks on real-world data", our temporal tokenization and reference frames "are both very useful additions, and are also well ablated", and our explicit motion modeling "makes this approach more usable for down-stream applications".

 

Multi-view video datasets DGS-LRM is designed as a feed-forward deformable Gaussian reconstruction method for monocular videos of dynamic scenes. As noted in the limitations section, our model, trained on continuous video with temporal tokenization, has difficulty generalizing to discrete image frames that simulate unnatural camera movements. As highlighted in DyCheck [1], teleporting among cameras using multi-view dynamic video datasets reaches unnatural speed far from real-world distribution. We leave developing a framework that can handle both continuous and discrete camera poses as future work. We also hope to highlight that DyCheck is the most widely used dataset for recent dynamic novel view synthesis from a monocular video.

 

Comparisons with static LRM methods Thank you for the constructive suggestion! Most state-of-the-art static feed-forward Gaussian reconstruction methods, such as GS-LRM [2], are not open-sourced. Therefore, we re-implemented GS-LRM to train a static baseline on the RealEstate10K dataset. This baseline achieved a PSNR of 28.77 on the RealEstate10K testing set, slightly higher than the PSNR of 28.10 reported in the original paper. The novel view synthesis quality on the Dycheck dataset, summarized in the table below, is significantly worse compared to DGS-LRM. We agree that this experiment underscores the value of explicitly modeling a deformation field. We will include this experiment in a revised version of the paper.

 

Synthetic data for training We find synthetic data crucial for the following 2 main reasons.

• Dual view supervision enhances novel view synthesis quality, especially with larger baselines.
• Direct supervision of the deformation field is necessary for accurate 3D tracking. Without it, rendering loss alone is insufficient for precise scene flow reconstruction.

Our method is among the first to address this challenging problem. Future work could involve more effective use of real data by distilling knowledge from foundational models, such as monocular depth and flow prediction.

 

Over-complete scene representation Thank you for the insightful feedback! We acknowledge that the per-pixel Gaussian is an over-complete scene representation for the foreground. In our current setting, we have not observed a significant impact on training and rendering speed. However, we anticipate that this issue may become more noticeable with longer context windows. Recent work [3] addresses this problem by utilizing opacity values to filter Gaussian points. We leave exploring this approach as future work.

 

Typos Thank you for pointing out! We will correct them in a revised version.

Table 2 We apologize for any confusion. Table 2 already includes the tracking numbers for the baselines.

 

[1] Hang, Gao, et al. "Monocular Dynamic View Synthesis: A Reality Check", NeurIPS 2022.

[2] Zhang Kai, et at. "GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting", ECCV 2024.

[3] Ziwen, Chen, et al. "Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats." arXiv preprint arXiv:2410.12781 (2024).

We thank the reviewer for their constructive feedback and recognition of our work. We appreciate their noting that our representation, combined with the key-frame rendering strategy, is "highly novel," and that our feed-forward method succeeds in "significantly improving speed while maintaining good quality."

 

Reproducibility We thank the reviewer raising the interest of our work and the impact of the datasets, which we deeply agree as well. We are preparing the release of our code, model and dataset. We are pending formal legal review to confirm the promise but we are working hard towards achieving this goal.

 

Fixed rotation and opacity For rotation, the per-pixel deformable 3D Gaussians are densely distributed in the foreground area. Each Gaussian typically has a small covariance. Therefore, we empirically did not observe any impact from fixed rotation. For opacity, our pilot study indicates that allowing changes in opacity enables the model to "fake" dynamic appearances without accurately predicting the deformation field, which negatively impacts tracking accuracy.

 

Frame attention Our preliminary experiments do not show significant improvements with Frame Attention under DGS-LRM's setting. We assume this is because our method's inputs include camera poses. But we also observe that Frame Attention is very effective in pose-free setting.

 

Keyframes and non-keyframes Thank you for the insightful suggestion! We conducted a quantitative comparison of view synthesis quality for keyframes and non-keyframes respectively on the Dycheck dataset. We observe a moderate degradation on non-keyframes compared to keyframes, which demonstrates the generalization ability of our method. We will include these results in a revised version of the paper for more comprehensive experiments.

Table 1 Quantitative comparisons of view synthesis quality for keyframes and non-keyframes on the Dycheck Dataset

 

Large viewpoint changes and long video For viewpoint changes, as mentioned in the limitations, our view synthesis quality deteriorates as the viewpoints deviate from the input trajectory. This is also true for other view synthesis methods. To alleviate this issue, we leverage dual view supervision from synthetic data and temporally distance reference frames as extra inputs. Both the qualitative results in Figure 4 and the quantitative results in Table 1 demonstrate that DGS-LRM achieves better quality compared to the SOTA feed-forward method and some optimization-based methods on the Dycheck dataset, even with large viewpoint changes.

For long videos, while we design flow chaining method to connect the scene flow from 2 sets of frames, it still causes a noticeable jump in appearance. Further works with a larger context window is required to address this challenge, such as an architecture similar to long LRM [1]. We will provide more detailed explanations in the limitations section of our revised paper.

 

Ablation studies We empirically find that further increasing the model size does not significantly reduce the training loss. Due to constraints on training time and cost, it is challenging for us to train multiple versions with varying model sizes. Our current model has 323 million parameters. Training at a batch size of 4 per GPU takes 80GB of VRAM. We fully agree that verifying whether DGS-LRM follows the scaling law is an important research problem, and we leave this for future work.

[1] Ziwen, Chen, et al. "Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats." arXiv preprint arXiv:2410.12781 (2024).