Section 4 Response to Reviewer 2xXk

We thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.

4.1 Questions

4.1.1 Failure case in DyNeRF dataset

Thank you to the reviewer for the constructive feedback. As discussed in our response to reviewer 4hTV, the failure on the DyNeRF dataset primarily stems from incorrect computation of the camera flow. More specifically, this issue arises from inaccurate depth estimation. As illustrated in Figure D of the PDF attachment, our method renders incorrect depth, leading to noticeable artifacts. Theoretically, removing the camera refinement step and directly deriving the Gaussian flow from the optical flow can avoid introducing erroneous depth. Unfortunately, after conducting experiments as suggested by the reviewer, the rendered scenes still exhibit certain artifacts, which typically appear only in the test views and not in the training views. We speculate that this issue arises because of the sparse viewpoint setup in the DyNeRF dataset, which is composed of videos captured by several fixed-position cameras rather than a single monocular video. These sparse viewpoints may lead to suboptimal Gaussian initialization and cause the model to overfit to the training views during subsequent optimization.

Nevertheless, we sincerely thank the reviewer for this insightful ideas, which inspired us to think more deeply about the possible limitations of our method. For scenes with stationary cameras, using reliable depth priors to provide regularized constraints may be a potential solution to alleviate this ill-posed problem, such as the recent work MoDGS[10]. In future work, we aim to combine sparse-view 3DGS methods to enhance the quality of dynamic scene reconstruction under sparse viewpoint conditions.

Section 3 Response to Reviewer pL6e

We thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.

3.1 Weaknesses

3.1.1 Novelty

We appreciate your detailed review of our work. We speculate that there might be some misunderstandings about the novelty and contributions of our work, and we'd like to make further clarification.

Our primary contribution is the separation of camera motion and object motion in optical flow, which provides clear motion guidance for dynamic Gaussians. Specifically, our method begins by acquiring an optical flow prior through an off-the-shelf optical flow estimation network. It then calculates the optical flow caused solely by camera motion by integrating depth and camera pose. Finally, it isolates the optical flow attributed to object motion to constrain the deformation of 3D Gaussians. This approach effectively creates a precise correspondence between the 2D motion prior and the 3D Gaussian motion.

Therefore, it is not entirely accurate to characterize our work as merely applying optical flow supervision to Gaussian motion. As shown in Table 3 of our main paper, directly constraining the Gaussians and deformation fields with optical flow even results in performance degradation. We believe this is because the camera motion incorrectly influence the deformation of dynamic Gaussians, while our method effectively addresses this issue.

Previous research on motion in NeRF and 3DGS fields, such as [7] and [8], have not addressed the interplay between camera and object motion from the perspective of optical flow formation. In contrast, our method first proposes to decouple optical flow for motion guidance, presenting a novel solution for dynamic scene rendering.

3.1.2 Performance gains

We appreciate the reviewer's examination of our work. We would like to provide a detailed clarification and further explanation regarding our performance contributions. Previous dynamic Gaussian methods often struggle to achieve accurate reconstruction in dynamic scenes with complex motion and imprecise poses, sometimes even failing to recover the structure of dynamic objects, as illustrated in Figure 6 of our main paper. By incorporating both motion guidance and camera pose optimization, our approach achieves more accurate reconstruction in complex dynamic scenes, as demonstrated in Table 1 and 2 of our main paper (with a 0.93dB mean PSNR increase in NeRF-DS and 2.3dB mean PSNR in HyperNeRF). The ablation study also indicates that our method significantly improves the rendering quality in scenarios with complex motion and inaccurate poses.

3.2 Questions

3.2.1 Modeling reflections

We appreciate the reviewer's questions. Indeed, we do not explicitly model reflections like NeRF-DS does. However, our approach has two advantages over NeRF-DS:

• Effectiveness of the deformable 3DGS Framework:

The 3DGS technique represents the scene as a set of anisotropic 3D Gaussians and employs an efficient differentiable rasterizer for rendering, achieving high-quality and real-time results [9]. On multiple datasets, 3DGS has matched or surpassed the performance of previous state-of-the-art NeRF methods. Our baseline method extends the rendering quality advantages of 3DGS to dynamic scenes by incorporating deformation fields into the 3DGS framework. As shown in Table 1, even our baseline method achieves rendering performance comparable to NeRF-DS methods without explicit modeling of reflections. The anisotropic 3D Gaussian spheres and the deformation field can model complex geometric details over time, enabling the baseline method to deliver high-quality rendering in dynamic scenes.

• Effectiveness of Motion Guidance:

While the baseline method delivers satisfactory rendering in most cases, there are still limitations in some challenging dynamic regions (e.g., the plate in Figure 5 of our main paper). Our method enhances performance in dynamic scenes by providing reliable motion constraints to the Gaussian deformation. By accurately separating and constraining different motion components in dynamic scenes, our method performs exceptionally well even in foreground areas with complex textures and reflections.

3.2.2 Training time

We appreciate the reviewer's question. We recognize that training time is a crucial factor in assessing the practicality of a method. Below, we provide detailed data of training times for our method on the NeRF-DS dataset:

Training Time on the NeRF-DS Dataset

Our method exhibits a slight increase in training time compared to the baseline method. This is primarily due to the inclusion of a differentiable Gaussian flow rasterizer[8] and the process of camera pose refinement. Notably, since the proposed improvements only influence the training process, our method maintains same real-time rendering speeds with our baseline during inference.

Section 2 Response to Reviewer 4hTv

We thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.

2.1 Weaknesses

2.1.1 Limiting assumptions

Please see Section 0.1 for details.

2.1.2 Beyond the limitation of prior work

We appreciate the reviewer's careful examination. Indeed, in some challenging scenarios, our baseline method is confined to reconstructing static backgrounds and exhibits limited performance on dynamic objects. As shown in Figure 6, the baseline method fails to capture even the basic structure of the broom. The baseline method also acknowledges this limitation and attributes it to inaccuracies in camera poses. Therefore, we propose camera pose refinement module to iteratively optimize the 3D Gaussians and camera poses. As a result, MotionGS is more capable of handling scenes with complex motions and inaccurate camera poses than our baseline method.

2.1.3 The quality of camera pose estimation

Please see Section 0.3 for details.

2.1.4 Ablation study in Table 5

We appreciate the reviewer's detailed analysis of our work. Here, we provide a detailed explanation and clarification regarding the ablation study presented in Table 5:

1. The Role of the Motion Mask:
   • The motion mask plays a critical role in our pipeline. As discussed in Section 4.2 of the main paper, our method benefits more from the motion mask than previous approaches[7]. Specifically, our method effectively utilizes the motion mask to filter out unreasonable motion flow in static areas. In contrast, when only optical flow supervision is used, the motion mask cannot be directly applied.
2. Limitations of Monocular Depth Estimator:
   • Our ablation experiments show that using an off-the-shelf monocular depth estimator introduces a performance drop. We argue that this is due to the scale ambiguity of monocular depth estimators, which results in inaccurate camera flow and motion guidance. In contrast, depth maps rendered by our method provide accurate scales and better details, as shown in the Figure B of the PDF attachment.
3. Impact of Optical Flow Networks:
   • The performance drop observed when switching optical flow networks is indeed unexpected. With further analysis, we find that FlowFormer performed inadequately in some challenging scenes, especially for the "plate" scene, resulting in an overall decrease in performance. Since GMFlow is trained on more datasets, it may have stronger zero-shot generalization capability than Flowformer.

2.2 Questions

2.2.1 Related work on optical flow decoupling

Please see Section 0.2 for details.

2.2.2 Fallback to start the iterative optimization

Thank you for this important question. Like our baseline method, we freeze the deformation field during the Gaussian initialization to obtain a reliable canonical representation. In most tested datasets, this canonical representation can be initialized well. Even with inaccurate camera poses, our method can render satisfactory results through iterative optimization. Currently, we have not yet explored extreme cases (e.g. COLMAP failure) or developed alternative initialization strategies to replace COLMAP.

2.2.3 Optical flow network

Thanks for your question. The GMFlow model[4], is trained on a range of datasets (KITTI, HD1K, FlyingThings3D, and Sintel). We use the pre-trained model from the original paper without further fine-tuning on our datasets.

2.2.4 Self-supervised optical flow methods

Thank you for the valuable suggestion. We used the self-supervised optical flow estimation algorithm MDFlow[5] in our ablation experiments. The results show that the motion constraints provided by the self-supervised optical flow estimation network cannot bring effective improvement to the network.

Novel View Synthesis Results of NeRF-DS Dataset

We speculate that this is because self-supervised methods cannot provide sufficiently accurate optical flow priors. With the increasing amount of annotated data and the development of foundational models in recent years, the generalization ability of fully-supervised optical flow estimation methods has significantly improved, surpassing traditional self-supervised methods. Similarly, works like MonoNeRF[6], Dynpoint[3], and DynIBaR[7] also use fully-supervised optical flow networks for optical flow priors.This phenomenon may also illustrates the importance of accurate motion priors, while erroneous or noisy motion constraints may even have a negative effect on the optimization.

2.3 Limitations

We appreciate the reviewer's insightful feedback. We recognize that the term "instability in optical flow computation" mentioned in our paper might have introduced some confusion. Upon further analysis, we find that the fixed and sparse camera viewpoints in the DyNeRF dataset hinder accurate depth rendering, affecting subsequent camera flow calculations and leading to artifacts as shown in Figure B of the PDF attachment. The inaccuracies in motion flow primarily comes from the inaccuracy of the camera flow, rather than a failure of the optical flow estimation itself. It is also important to clarify that the DyNeRF dataset is not continuous monocular video but rather dynamic scenes with sparse viewpoints, which posed challenges to the canonical 3D Gaussian initialization. This poor initialization prevents our method from rendering accurate depth and performing subsequent optimizations, and in future work we consider combining sparse-view 3DGS methods to improve the robustness of our MotionGS.

Section 1 Response to Reviewer rwSL

We thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.

1.1 Weaknesses

1.1.1 Time, memory, and storage costs

Thank you for your question. In Table 4 of Appendix A.2, we have provided a detailed breakdown of the rendering speed and storage requirements of our method on the NeRF-DS dataset. Our approach aims to introduce explicit motion constraints and pose refinement during training without bringing additional burden to the original Deformable-3DGS during inference. Thus, the rendering speed and inference time of our method are consistent with our baseline.

For model training, we list the training time per scene and peak memory usage on the NeRF-DS dataset as below, providing a comprehensive assessment of resource usage during training. Compared to our baseline, our approach incurs increased training time and peak memory usage. This is primarily due to the additional rendering of Gaussian flow and the refinement of camera poses, which are necessary for our method.

Training Time on the NeRF-DS Dataset

Max GPU Memory Usage on the NeRF-DS Dataset

1.1.2 Data sampling mechanism

Thanks for your question. Here is a more detailed explanation of our data sampling mechanism:

• Data Sampling Strategy: We adopt the same data sampling strategy as the baseline method, i.e., reading image sequences in a randomly shuffled order. For an N-frame video, the frames are shuffled and then read sequentially. In each iteration, we read two frames and caluate the optical flow between them. To enhance efficiency, the second image from the last iteration is used as the first image in the current iteration. Thus, except for the first iteration, only one new image is read in each subsequent iteration. Consequently, there are (N-1) iterations per epoch, with optical flow computed once in each iteration. This strategy balances the introduction of accurate motion priors with maintaining training efficiency.

• Optical Flow Calculation and Storage Strategy: During the first epoch of training, we calculate the optical flow for all adjacent frame pairs, resulting in a total of (N-1) optical flow maps. In subsequent epochs, we do not reshuffle the image sequence, allowing us to reuse the optical flow maps calculated in the first epoch. This effectively eliminates the need to recompute optical flow maps in each epoch, significantly reducing computational overhead. Taking the "as" scene (846 frames) in the NeRF-DS dataset as an example, the optical flow calculation for each pair takes approximately 22 ms, resulting in a total computation time of 18.7 seconds. Each optical flow map requires around 0.99 MB of storage, resulting in a total storage requirement of 835.13 MB.

1.2 Questions

1.2.1 Self-supervised flow loss

We appreciate the insightful suggestion provided by the reviewer. Following the reviewer's suggestion, we conduct experiments to evaluate this approach and have confirmed its effectiveness. Specifically, we estimate the Gaussian flow corresponding to the optical flow and use it to warp the $I_t$ frame. We then compute the photometric loss with the $I_{t+1}$ frame. In experiments on the NeRF-DS dataset shown as below, this method outperforms our baseline but is less effective compared to our proposed method.

Novel View Synthesis Results of NeRF-DS Dataset

We hypothesize that the discrepancy arises because the self-supervised loss may not provide accurate supervision in areas with similar colors. Nevertheless, it is evident that employing self-supervised optical flow loss can reduce dependence on off-the-shelf optical flow estimation. We provide qualitative experimental results in Figure A of the PDF attachment, which show that this idea can effectively provide motion constraints. When an optical flow estimation network is either unavailable or inaccurate, this approach can serve as a valuable alternative to improve rendering quality.

We sincerely appreciate your valuable idea, which has greatly enhanced our understanding to the problem. We are excited to incorporate this perspective into our research.