Thank you for your helpful comments! Our replies to your questions and revisions to the submission are stated below.

Weakness

W1: The innovation is quite incremental in comparison with closely related works like Deformable 3DGS and 4DGS.

A1:

• Difference between our MoDGS and baseline methods, Deformable 3DGS and 4DGS. As introduced in L036, Deformable-3DGS and 4DGS require multiview videos or "teleported" monocular videos as inputs, which is not a real monocular video setting. In contrast, MoDGS is able to reconstruct 4D fields from real casually captured monocular videos, which move smoothly and slowly and thus are significantly more challenging. Deformable-3DGS and 4DGS fail on these casually captured monocular videos as demonstrated in our experiments including videos (from 0m12s to 0m56s) and tables (Tab.1, Tab.6, and Tab.7).
• Our contributions. MoDGS contains two novel techniques, i.e. 3D-aware-init and ordinal depth loss.

1. Gaussian splatting can be initialized from SfM points in static scenes but how to robustly initialize dynamic Gaussian fields is not well-studied. We show that the previous initialization method adopted in dynamic 3DGS does not work well in the real monocular setting in our experiments(Fig.7 and Tab.2). Thus, we propose 3D-aware-init which provides a robust starting point for the deformation field and canonical space Gaussians.
2. In the real monocular setting, we have to rely on monocular depth estimation due to the weak multiview constraints in casually captured videos. However, how to supervise the dynamic Gaussian fields with inaccurate monocular depth maps is still an open question. The proposed ordinal depth loss is designed to effectively utilize the inconsistent depth for the 4D reconstruction.

W2: The approach relies heavily on pre-trained depth models. The approach leverages external models as priors, potentially limiting its novelty and raising questions regarding knowledge distillation. The extent to which these pre-trained models influence the final performance is not rigorously analyzed.

A2:

• Why rely on monocular depth? As introduced in L041, casually captured videos usually have minor camera motions, which provide insufficient multiview constraints. Thus, we have to introduce monocular depth estimators to constrain the geometry, which is also adopted by all concurrent works[1,2,3,4] that process casual monocular video.

• Is the proposed method robust to the noise of estimated depth? In Appendix A.8, we present an ablation study on robustness to depth quality by introducing Gaussian noise into the input depth maps. After adding Gaussian noise, the PSNR changed by less than 0.2, the SSIM by less than 0.008, and the LPIPS by less than 0.002. The results demonstrate that our method is robust to a certain range of noise and is not sensitive to variations in depth quality.

• Emerging trends in utilizing off-the-shelf models for specialized tasks. Last but not least, it is worth noting that in the era of deep learning, the performance of large models tailored for specific tasks has significantly improved due to the surge in data and computing power. Utilizing pre-trained off-the-shelf models as one component of a multi-stage processing system to be constructed is becoming a prominent and emerging trend.

W3: While MoDGS integrates external depth estimation for initialization, there is no formalized knowledge distillation to adaptively refine the model during training. This absence may reduce the adaptability of MoDGS across different dynamic scenes where pre-trained depth estimators may not perform equally well.

A3:

• How do we distill the monocular depth for supervision? We adopt the ordinal depth loss as the supervision loss to constrain the geometry from the estimated monocular depth maps.

• Is MoDGS robust to different depth estimation methods or noises in the depth estimation? Yes. The table above has proved the robustness to noise. We have also tested our method in combination with different depth techniques in Tab. 9, all of which yielded effective results compared with Pearson depth loss.

• Why our method is robust to noise in the depth estimation? As introduced in L305, the estimated depth maintains relatively stable orders, even though the absolute values can vary significantly. Our ordinal depth loss relies solely on the orders of pixel pairs. Moreover, our rendering loss will provide further corrective effects.

Q1: How does MoDGS handle scenarios where the pre-trained depth estimator provides inconsistent depth due to environmental variations? Has any analysis been conducted to measure performance stability when GeoWizard or other models are less reliable?

A1:

• The depth inputs indeed exhibit scale inconsistencies, and we did not assume these would be consistent as shown in the input depth videos of the supplementary demo video)

• In the 3D-aware-init stage, we address this inconsistency by estimating a rough scale for each depth map (see Sec 3.2, in Initialization of depth scales paragraph), which does not need to be very accurate.

• In the optimization stage, we did not directly apply L1 or Pearson correlation depth loss for direct supervision but adopted the ordinal depth loss because we observed that the depth orders are relatively more consistent.

Q2: Would MoDGS perform as well on datasets with higher motion complexity or less predictable scene geometry? Testing on a broader range of datasets, such as those with cluttered backgrounds or multiple moving objects, would better validate the method's generalization.

A2:

• Scenes such as deaddrift(from MCV dataset, results in Fig.6 and video starts at 3m20s), coffee_martini(from DyNeRF dataset, results in Fig.5 and video starts at 2m12s), and camel(from Davis dataset, results in Fig.11 and video starts at 3m44 ) are challenging due to their complex movements and cluttered backgrounds. Nevertheless, our method remains effective and produces reasonable results in these scenarios.
• In the playground and balloon2-2 scenes, which feature two moving objects (the person and the balloon) and complex motion, MoDGS handles the situation relatively well and generates reasonable results. In Appendix A.17, we present the NVS results on these two scenes and we will update the demo video in the final version.
• We agree with the reviewer that reconstructing 4D dynamic fields in complex scenes with just casually captured monocular videos is still a challenging task. Our work already makes significant improvements in this setting with moderate complexity, which could be the basis for addressing more cluttered scenes with more complex motions in future works.

Q3: Considering MoDGS’s reliance on single-view depth priors, would a formalized knowledge distillation framework improve model autonomy by adapting these priors dynamically during training?

A3:

We agree with the reviewer that our method can be regarded as a distillation framework from the perspective of knowledge distillation to get consistent video depth. In our framework, we adopt the 3D Gaussian field as the 3D representation and the ordinal depth loss as the supervision to distill the monocular depth estimation. Meanwhile, we adopt the rendering loss to further regularize the 3D representation, which enables us to distill temporally consistent monocular video depth from the inconsistent estimated depth maps. The inconsistent input depth and the distilled consistent video depth can be visualized in the supplementary video. We will add this discussion in the revision(Appendix A.15).

• [1] Liu et al., Robust Dynamic Radiance Fields, CVPR, 2023.
• [2] Lee et al., Fast View Synthesis of Casual Videos with Soup-of-Planes, ECCV, 2024.
• [3] Lei et al., MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, arXiv preprint, 2024.
• [4] Wang et al., Shape of Motion: 4D Reconstruction from a Single Video, arXiv preprint arXiv:2407.13764, 2024.

Dear Reviewer Gzks,

We sincerely extend our gratitude for dedicating your time and effort to reviewing our submission to ICLR. Your acknowledgment of our efforts is greatly valued. In response to your further feedback, we have conducted ablation studies employing three supplementary depth estimation methods, showcasing the robustness of our approach. We kindly invite you to review our detailed responses at your earliest convenience and let us know if there are any remaining concerns. We are eager to engage in further discussions on any outstanding issues.

Thank you once again for your insightful feedback.

Best regards,

The authors

Dear Reviewer Gzks,

Thanks for your valuable time reviewing our paper. The discussion session is closing soon. We are eager to hear your additional feedback of our paper. Thank you!

Best Regards,

The Authors

Dear Reviewer Gzks,

Thank you very much for the insightful discussion and for taking the time to review our work. We truly appreciate your efforts in helping us enhance our paper. If you have any further questions, we are more than happy to reply.

Best regards,

The Authors

We thank the reviewer for the effort in reviewing our paper. Our response to the all concerns is listed below.

Weakness

W1: The contributions and innovations are limited because this work is based on the canonical space paradigm of 3DGS combined with deformation fields, with the main contributions being a deformable 3DGS initialization method and a depth loss.

A1:

• Novelty in comparison with baseline methods with canonical space and deformation fields. Although baseline methods Deformable-3DGS and 4DGS also use canonical space and deformation fields, these baseline methods all require multiview videos or "teleported" monocular videos as inputs, which is not a real monocular video setting. In contrast, our MoDGS is able to reconstruct 4D fields from real casually captured monocular videos, which move smoothly and slowly and thus are significantly more challenging. Deformable-3DGS and 4DGS fail on these casually captured monocular videos as demonstrated in our experiments including videos (from 0m12s to 0m56s) and tables (Tab.1, Tab.6, and Tab.7).
• Our contributions. To address this challenge, MoDGS contains two novel techniques, i.e. 3D-aware-init and ordinal depth loss.

1. Gaussian splatting can be initialized from SfM points in static scenes but how to robustly initialize dynamic Gaussian fields is not well-studied. We show that the previous initialization method adopted in dynamic 3DGS does not work well in the real monocular setting in our experiments (Fig.7 and Tab.2). Thus, we propose 3D-aware-init which provides a robust starting point for the deformation field and canonical space Gaussians.
2. In the real monocular setting, we have to rely on monocular depth estimation due to the weak multiview constraints in casually captured videos. However, how to supervise the dynamic Gaussian fields with inaccurate monocular depth maps is still an open question. The proposed ordinal depth loss is designed to effectively utilize the inconsistent depth for the 4D reconstruction.

W2: The experimental comparisons lack fairness. In most quantitative comparisons, this work is only compared against methods that require multi-view camera input. And only found comparative results of RoDynRF presented.

A2:

• Choice of baseline methods. We have tried our best to include all available competitive baseline methods including real monocular video methods, i.e. RoDynRF and Shape-of-Motion (arxvi:2407.13764, details in Appendix A.6), and the teleported monocular video methods, i.e. Dynamic-GS and SC-GS. We have demonstrated improved rendering quality than all these baseline methods.

• Why not report the quantitative results of RoDynRF? Since the casually captured monocular videos only contain weak multiview constraints, all methods may have a different scene scale from the ground truth. We have tried to align the scale of RoDynRF with the ground truth but the alignment still fails in some cases. Thus, we provide the qualitative comparison (starting at 4m22s in the supplementary video), which shows that our method achieves much better rendering quality.

We would like to express our sincere gratitude for dedicating your time and effort to reviewing our manuscript. We have carefully considered and responded to all the concerns you raised in your review, as detailed in our response and the revised manuscript.

As the Reviewer-Author discussion phase approaches its conclusion, we kindly await any further feedback you may have. Should you have any additional questions or require further clarification, we would be more than happy to provide detailed responses.

Thank you once again for your valuable assistance.

Dear Reviewer FCsY,

Thanks for your valuable time reviewing our paper. The discussion session is closing soon. We are eager to hear your additional feedback of our paper. Thank you!

Best Regards,

The Authors

We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Weakness

W1. The paper may benefit from more discussion on its limitations. 1. The assumption of consistent depth orders among frames may not hold in scenes with complex occlusions or reflections. 2. MoDGS may face challenges in scenes with rapid or erratic movement. 3. The method depends on single-view depth estimators, inheriting their limitations in complex with specular surfaces or low-lit environments. A detailed analysis of estimator quality and integration with other methods could enhance adaptability.

A1:

• Assumption of consistent depth orders. We agree with reviewers that MoDGS assumes overall depth order consistency that can be satisfied by most depth estimators, GeoWizard, DepthAnything, etc. For these depth estimation methods, some flickering occurs in certain regions (as shown in our supplementary demo video), but the overall depth order remains consistent and thus our method successfully reconstructs dynamic Gaussian fields. Without this depth order consistency, our method could fail. We have added this discussion in the revision(L525).

• Challenges in rapidly changing scenes.

Rapidly moving scenes are indeed challenging for dynamic Gaussian field reconstruction. Such rapid motions may bring an inaccurate camera pose estimation, which is still challenging for MoDGS and all existing dynamic Gaussian reconstruction methods. We may combine the deblur methods with some motion prior to solving this in future works. Following your suggestions, we have added this limitation to the revised version(L527).

• The method depends on single-view depth estimators, inheriting their limitations in complex with specular surfaces or low-lit environments.

We agree with the reviewer that specular surfaces and low-lit conditions pose challenges for monocular depth estimation. In our experiments, we demonstrate a certain level of robustness in handling specular and low-lit scenes. For instance, MoDGS successfully reconstructs the windows in the Cook Spinach scene (from the DyNeRF dataset, with the video starting at 2m19s, which is also a low-lit scene), the windshield in the Truck scene (from the NVIDIA dataset, video starting at 2m54s), and the balloon in the Balloon1-2 scene (from Nvidia dataset, video start at 2m43s). However, for severely low-lighting or specular scenes, our method could fail due to the absence of reasonable depth estimation and we have added this in the limitation(L528).

Question

Q1: How about Comparing ordinal depth loss with other depth loss functions, such as perceptual depth consistency? what were the observed advantages or disadvantages?

A1:

Thanks for your suggestion. We present the results of this ablation study with perceptual (LPIPS) depth loss in Appendix A.16. Such LPIPS loss produces much better results than the vanilla Pearson loss, which demonstrates that such LPIPS loss is also insensitive to the absolute difference between depth values in some sense. Meanwhile, compared to the depth maps generated using the LPIPS loss, our maps are noticeably smoother and exhibit less noise. The reason may be that LPIPS loss is mainly trained on RGB images and performs less discriminative on the images converted from depth maps.

Q2: How robust is MoDGS in scenarios with heavy occlusions or specular reflections? Would integrating additional priors or multi-scale depth estimations help in such cases?

A2:

• How to handle scenarios with heavy occlusions. We agree that MoDGS may fail to handle scenarios with heavy occlusions. If the occluded regions are observed at some timestamps, MoDGS is able to accumulate information among different timestamps to complete these occluded regions on some timesteps as shown in Appendix A.4. However, MoDGS cannot generate new contents for the occluded regions, which indeed degenerate the rendering quality on occluded regions. Incorporating some generative priors such as diffusion models could alleviate this problem.

• Possible solutions dealing with specular scenarios. MoDGS can reconstruct some specular objects but show some artifacts because MoDGS does not involve any special design for specular objects. We have added this discussion in the revision(L528). As shown by recent work [4], a potential solution for this is to introduce inverse rendering in dynamic scenes, which enables accurate modeling of specular surfaces in dynamic scenes.

Q3: How about comparing with recent depth consistency techniques, particularly those used in self-supervised monocular depth estimation?

A3:

• Thank you for your suggestion. Our method is mainly targeted for the novel-view synthesis while we agree that MoDGS produces consistent video depth maps as a side product. Recent methods like [1,2,3] produce consistent depth maps but are targeted for depth estimation. We will conduct an experiment to compare the depth map quality of MoDGS with these baseline methods in the revision and we are still working on it. We will update the results once finished.

[1] Luo et al., Consistent video depth estimation, ToG, 2020.

[2] Wang et al., Neural Video Depth Stabilizer, ICCV, 2023.

[3] Xu et al., Depthsplat: Connecting gaussian splatting and depth, arXiv preprint, 2024.

[4] Fan et al., SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes, arXiv preprint, 2024.

We would like to express our sincere gratitude for dedicating your time and effort to reviewing our manuscript. We have carefully considered and responded to all the concerns you raised in your review, as detailed in our response and the revised manuscript.

As the Reviewer-Author discussion phase approaches its conclusion, we kindly await any further feedback you may have. Should you have any additional questions or require further clarification, we would be more than happy to provide detailed responses.

Thank you once again for your valuable assistance.

Hi! Regarding Q3, we have finished the comparison with a recent work the neural video depth stabilizer (NVDS) [1]. The results are added in manuscripts (A.15, Fig 16) and the video in supplementary files (from 4m41s to 4m52s). It can be observed that both depth maps are temporally consistent, but the depth maps of MoDGS contain more details than NVDS.

• [1] wang et al., Neural Video Depth Stabilizer, ICCV, 2023.

We thank the reviewer for the positive feedback and constructive suggestions. Our response to the reviewer’s concerns is below:

Weakness

W1:Potential limitations in generalizing this approach to different depth estimation models? MoDGS is validated across several datasets, which demonstrates its robustness. However, the paper could discuss the potential limitations in generalizing this approach to different depth estimation models. It would demonstrate the robustness of the proposed method and its generalizability.

A1:

We agree that the proposed method may fail when the prior depth estimation methods completely fail because casually captured monocular videos contain fewer multiview constraints and thus we rely on the depth prior to constrain geometry. However, current monocular depth estimators, like DepthAnythingv2, Depth Pro, GeoWizard, and Marigold, are powerful and can produce reasonable depth maps in most cases. We have added this discussion to the paper(L525) according to your suggestion.

To further prove the robustness to different depth quality, in Appendix A.8, we present an ablation study on robustness to depth quality by introducing Gaussian noise into the input depth maps. The results demonstrate that our method is robust to a certain range of noise and is not sensitive to variations in depth quality. In Appendix A.11, we present an ablation study on the robustness to different depth estimation methods, it can be observed that with more stable video inputs, our ordinal depth loss still performs good results, which is better than the Pearson depth loss.

Question

Q1:Suggestion about changing to a more informative title.

A1:

Thank you for your suggestion. We have changed it to "MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors".

We would like to express our sincere gratitude for dedicating your time and effort to reviewing our manuscript. We have carefully considered and responded to all the concerns you raised in your review, as detailed in our response and the revised manuscript.

As the Reviewer-Author discussion phase approaches its conclusion, we kindly await any further feedback you may have. Should you have any additional questions or require further clarification, we would be more than happy to provide detailed responses.

Thank you once again for your valuable assistance.

Hi! Regarding W1, We conducted ablation studies using three additional depth estimation methods[1,2,3] and compared the performance of our MoDGS. The results are presented in Appendix A.20. When we switched to different depth estimation methods, the PSNR varied by less than 0.35, the SSIM by less than 0.01, and the LPIPS by less than 0.015. These minimal changes demonstrate the robustness of our approach to varying distributions of real-world depth estimators. The main reason is that our method only relies on the order of depth values and most depth estimators are able to estimate correct depth orders in spite of their varying error types in different scenes. However, we also agree that the strong varying environments or cluttered scenes may break the correctness of predicted depth orders so our method may fail in these extremely challenging cases.

Some qualitative visualization of depth inputs from different methods and depth maps rendered from MoDGS are added in manuscripts (A.20, Fig 17), and supplementary video (from 4m52s to 5m08s). It can be observed that MoDGS produces consistent video depth from different types of estimated depth maps.

• [1] Ke et al., Repurposing diffusion-based image generators for monocular depth estimation, CVPR, 2024.
• [2] Yang et al., Depthanything: Unleashing the power of large-scale unlabeled data, CVPR, 2024.
• [3] Shao et al., Learning Temporally Consistent Video Depth from Video Diffusion Priors, arXiv preprint, 2024.

Our revisions summary:

Regarding our manuscripts, We newly added one new section in the Appendix and updated experiments results in A.15. Regarding our demo video in supplementary files, we added two sections.

Manuscripts:

• In A.15, we visually compare rendered depth maps using our MoDGS with stabilized depth maps from a recent video depth stabilization method.
• In A.20, We present quantitative and qualitative results using different depth estimation methods, demonstrating our method's robustness.

Demo video in supplementary:

• From 4m42s to 4m52s, we present visual comparison of rendered depth maps with a video depth stabilization method.
• From 4m52s to 5m08s, we showcase a visual comparison using four different depth estimation methods.