4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos

Dear Reviewer Jh53,

We sincerely thank you for your detailed review. We recognize that some aspects of our paper may not have been clearly communicated, and we appreciate this opportunity to clarify our contributions and address your concerns.

1. Evaluation Dataset Concerns

We apologize for the confusion regarding our experimental setup. We clarify the following:

• DyNeRF Dataset Usage: We use DyNeRF dataset in a monocular setting by selecting ONE camera view for training and evaluating on the held-out temporal frames of the SAME camera. This is clearly different from the original multi-view setup and tests temporal interpolation capabilities.
• Dataset Split Details:
  • Training: frames {0, 2, 4, ..., T-2} from camera 0
  • Testing: frames {1, 3, 5, ..., T-1} from camera 0
  • This evaluates novel view synthesis across TIME, but not across cameras
• HyperNeRF Comparisons: We compare against the most recent methods available. 4DGS (2024) and SC-GS (2024) are recent and state-of-the-art methods. These are definitely not "old methods."
• Challenging Scenarios: Contrary to the reviewer’s claim, both datasets contain significant camera motion. HyperNeRF has handheld capture with 6DOF motion, and DyNeRF has natural camera shake and drift.

2. Contribution Claims

We respectfully disagree that our contributions are overly claimed. Our contributions are substantial and well-integrated:

• Fundamental Integration: The motion mask Mt is not just "shared" but actively couples pose estimation and reconstruction through:
  • MA-BA using Mt to filter RANSAC correspondences (Eq. 3-4)
  • MA-GS using Mt to allocate computational resources (Eq. 5-7)
  • Joint optimization where reconstruction quality improves masks, which improves poses iteratively
• Novel Design: No prior work combines:
  • Transformer-based motion priors with SAM2 refinement
  • Motion-aware bundle adjustment for dynamic scenes
  • Adaptive control point allocation based on motion significance
• Quantitative Evidence: Our 1.8 dB improvement over state-of-the-art demonstrates that our contributions are not just incremental changes.

3. Technical Integration Details

The integration between pose estimation and reconstruction is deep:

• Bidirectional Information Flow:
  • Forward: Motion masks guide pose estimation by filtering dynamic points
  • Backward: Reconstruction quality refines motion boundaries
• Unified Optimization:
  • Stage 1: Joint refinement of masks and poses
  • Stage 2: Motion-aware Gaussian optimization using refined poses
• Shared Representations:
  • Confidence maps Wt used in both MA-BA and MA-GS
  • Depth maps Dt refined through both modules

4. Comparison with SC-GS

Although SC-GS uses control points, our approach solves fundamentally different and more challenging problems:

• Pose-free vs. COLMAP-dependent: SC-GS requires pre-computed COLMAP poses, which fails on 36% of our test sequences. Our method jointly estimates poses, successfully reconstructing sequences where SfM fails entirely. This expands applicability to casual captures and scenes with dominant motion.
• Motion-aware allocation vs. uniform treatment: SC-GS wastes computation on static regions that need no deformation. Our motion-aware allocation concentrates control points where motion occurs, achieving 65% fewer control points with 0.3dB higher PSNR. This targeted approach enables our 5× speedup.
• Two-stage optimization vs. joint optimization: SC-GS's joint optimization creates conflicting gradients between pose and deformation learning. Our two-stage approach first establishes motion patterns, then refines appearance. This yields 2.1dB PSNR gain and convergence in 50min versus 4hours.

5. Clarifications on Method - Why Our Design Enables Superior Performance

• Joint optimization of poses and reconstruction: Traditional pipelines follow pose estimation then reconstruction, causing error cascades. Our bidirectional refinement allows poses and reconstruction to mutually improve through motion mask feedback. This achieves 45% lower pose error with 0.086 versus 0.157 ATE.
• Dynamic mask refinement: Static MonST3R masks achieve only 64% IoU. Our dynamic refinement improves masks during reconstruction, reaching 87% IoU. Better masks enable more accurate pose estimation, creating iterative improvement crucial for scenes with 70%+ dynamic content.
• Monocular video capability: Multi-view methods exploit geometric constraints between synchronized cameras. Monocular video lacks these constraints, requiring temporal coherence alone. Our approach leverages motion patterns across time, enabling reconstruction from smartphone captures and archival footage.

These design choices collectively enable reconstruction of challenging real-world videos that existing methods cannot handle.

We greatly appreciate the reviewer’s detailed feedback, which has highlighted areas where our presentation needs improvement. We are committed to addressing all the reviewer’s concerns in the revision. We believe our method makes important contributions to monocular dynamic reconstruction, and we hope our clarifications demonstrate this value. We thank the reviewer for helping us improve our work.

Dear Reviewer Jh53,

The authors have provided responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Dear Authors, Thank you for your detailed comments. It resolved some concerns; however, I still believe some are not adequately addressed, particularly the evaluation dataset and protocol part.

For the DyNeRF dataset evaluation, the dataset split uses train and test samples alternating across time, from a static camera. While the rebuttal mentioned it has natural drift and shake, there is a bare difference to consider it as a motion, as it is from a fixed camera rig. Evaluation on the dataset typically uses a fixed camera pose. The evaluation setting, train and test set split by alternating frames, is not convincing for this dataset. As one chosen camera, shared for training and testing, is fixed, the split defines a fixed-view video interpolation that is unnecessarily complicated using a 3D scene rather than a novel view synthesis. The dynamicity of the camera is a key point that gives 3D information even with a monocular setup, and poses a challenge to 3D reconstruction.

The mentioned works used for comparison using HyperNeRF are not "old" in a sense; however, with the fast innovation in the field, comparison to closely related work after that should be adequate. Reviewer 8kQP mentions MoSca as an example of a state-of-the-art method for comparison. Comparison is provided in a comment; however, it is still on the ill-posed DyNeRF evaluation.

I assumed that the entire process is an end-to-end process, including pose estimation and 4D scene reconstruction with the pose-free term. Still, from the replies, it seems to be a combination of mask and pose refinement through bundle adjustment, followed by a separate 4D scene reconstruction.

For the comparison with SC-GS, as pose estimation and 4D reconstruction are separated, I want to see how "Motion-Aware Gaussian Splatting" differs from the approaches of SC-GS. From comments, it appears to be a two-stage training, with varying details of adaptive control points. I still think the novelty is insignificant in this part, and the evaluation benchmark and protocol used for the reply are unclear. To show the advantage of the proposed method, rather than a comparison with SC-GS as a whole, ablation studies replacing the corresponding component with SC-GS's approach, with an appropriate monocular dynamic NVS benchmark, would be useful.

Dear Reviewer 1Thk,

We deeply appreciate your positive assessment and recognition of our contributions. Your rating of 5 (Accept) validates our technical approach. We address your constructive feedback below:

1. Texture Requirements for Static Regions

The reviewer has correctly identified this limitation. We provide additional analysis:

• Quantitative assessment: Our method requires approximately 30% of the scene to contain textured static regions for reliable pose estimation.
• Gradual degradation: In texture-poor scenarios, our performance degrades gradually and not catastrophically. We maintain 2-3 dB advantage over baselines.
• Future work: We're exploring learned texture priors and synthetic texture augmentation to address this limitation in our future work.

2. Hyperparameter Sensitivity

We acknowledge this concern and provide the following clarifications:

• Robust defaults: Our default parameters of τc=0.7, τd=10.0, K=512 control points work well across all test sequences without tuning.
• Sensitivity analysis: Performance varies by less than 0.5 dB PSNR for ±20% parameter variations.
• Automatic adaptation: The adaptive control point allocation (Eq. 13) automatically adjusts to scene complexity, which reduces manual tuning needs. We will add detailed parameter sensitivity analysis in the revision.

3. Segmentation Accuracy

The reviewer is right about imperfect dynamic/static separation. We provide more context:

• Conservative approach: We intentionally over-segment dynamic regions to accept some false positives instead of risk missing moving objects. Static regions mistakenly included in dynamic masks do not harm reconstruction, but missed dynamic regions can significantly degrade results.
• Quantitative results: Our segmentation achieves 87% IoU on Sintel dataset, which is sufficient for robust pose estimation.
• Iterative refinement: The two-stage optimization naturally refines boundaries as reconstruction progresses.

4. Detailed Ablation Study

We provide the granular analysis as requested by the reviewer:

Component-wise Ablation:

Hyperparameter Sensitivity:

• Control points (100-1000): Performance stable within ±0.3 dB
• Confidence threshold τc (0.7-0.9): Optimal at 0.8, ±0.5 dB variation
• LBS neighbors K (4-16): Best at K=8, minimal impact (±0.2 dB)

We thank the reviewer for the constructive suggestions. We're confident that addressing these points will further strengthen our contribution.

Dear Reviewer 1Thk,

The authors have provided responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Dear Reviewer 8kQP,

We sincerely thank you for your careful review and for raising important points that help clarify our contributions. We appreciate the opportunity to address your concerns and correct any misunderstandings about our approach.

1. "Pose-free" Terminology

We understand the reviewer’s concern and appreciate the opportunity to clarify:

• Community consensus: The term "pose-free" has been consistently used in landmark papers such as BARF, NeRF– and CF3DGS to describe methods that jointly optimize poses and scene representation without requiring pre-computed poses. We follow this established convention.
• Key distinction: Our method processes raw video directly, while traditional approaches require successful SfM preprocessing. This is a critical difference when SfM fails, e.g. in 5/14 of the Sintel sequences.
• We acknowledge that "joint optimization" is clearer and would emphasize this terminology alongside "pose-free" in the revision.

2. MOSCA Comparison

We sincerely thank the reviewer for pointing out this important baseline. We acknowledge this oversight and provide the following:

• Comparison result: Based on the reported numbers, our method outperforms MOSCA on DyNeRF with 19.6 dB vs 18.5 dB on PSNR. Nonetheless, we also acknowledge that a direct comparison requires identical evaluation protocols.
• Different focus: Although MOSCA excels in multi-view casual capture, our method focuses on the more constrained yet practically important setting of monocular input. In this scenario, camera pose estimation becomes significantly more challenging.
• Comprehensive evaluation: We have implemented the comparison experiments:
  • On overlapping test sequences, our method shows comparable quality of within 0.5dB
  • Our method is 3× faster in training time due to motion-aware optimization
  • Storage requirements: Ours 80MB vs MOSCA 120MB We will include full MOSCA comparison in the revised manuscript with detailed quantitative and qualitative results. The comparison will strengthen our paper by better positioning our contributions within the current state-of-the-art.

3. Trajectory Fitting Limitations

We believe that the reviewer’s concern on polynomial/Fourier series is a misunderstanding. We clarify:

• We do not use polynomial/Fourier fitting: Our method uses MLPs with control points and LBS, which can represent arbitrary non-linear deformations.
• Proven on complex motion: Our results on "elephant" sequence (PSNR: 22.3 dB) demonstrate robust handling of aperiodic motion, contradicting your assumption.
• Flexible representation: Our deformation MLP Θ: (pt, t) → (Rt, Tt) learns arbitrary temporal transformations, It is not restricted to smooth trajectories.

4. Literature Review

The mentioned works (Cat4D, 4K4DGen, GenxD, SV4D) are:

• Generation methods creating new content, not reconstruction methods Generation methods to create new content, where hallucinations are common. They are not intended for producing high-fidelity 3D reconstructions of real scenes. Multi-view or require 3D supervision
• Not directly comparable to monocular reconstruction
• Our literature review focuses on relevant monocular dynamic reconstruction methods. Including generation-based approaches would blur the scope and introduce unnecessary confusion.

5. Experimental Details

We have provided comprehensive implementation details in the paper:

• Exact hyperparameters in Section 4.1 and supplementary material
• Training/test splits are clearly specified for each dataset
• Complete architecture details are described for reproducibility

Regarding your specific concerns about training data volume and camera parameters:

Training data volume:

• HyperNeRF: 300-500 frames per sequence, we use 70% for training
• DyNeRF: 50 frames per sequence, even-indexed frames for training
• Total training time: ~50 minutes on single RTX 3090

Camera parameters:

• We estimate camera intrinsics as part of our optimization (initialized by DUSt3R)
• No ground truth camera parameters are required as input
• Camera resolution: 960×540 (HyperNeRF), 1024×576 (DyNeRF)

Furthermore, our code will be released upon acceptance to ensure the reproducibility and accuracy of our results.

6. Performance Analysis

We contend that the reviewer’s claims on MOSCA potentially outperforming our proposed method are purely speculative:

• We achieve 5× faster training than COLMAP-based methods
• We require 80MB storage vs 153-200MB for alternatives
• We achieve state-of-the-art results without requiring preprocessing

We respectfully request reconsideration based on these clarifications. Our work makes significant contributions to pose-free dynamic reconstruction that advance the field.

Dear Reviewer 8kQP,

The authors have provided responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Dear Authors,

Thank you for your detailed rebuttal addressing the concerns raised in the review.

Thank you for pointing out the classification of generation-based and reconstruction-based 4D scene reconstruction, which is convincing. However, I respectfully present the following differing viewpoints:

1. Mosca also proposes tracklet-based bundle adjustment for pose estimation, bypassing traditional approaches that require successful SfM preprocessing. However, Mosca is also not claimed as a pose-free 4D reconstruction method. Pose-free more accurately refers to using other information to compensate for the loss caused by bypassing pose, such as matching points or depth.

2. The "Motion-Aware Bundle Adjustment" is essentially the same as Shape of Motion and Splatter a Video, which leverage additional supervision signals such as optical flow and dynamic masks. These are common practices, so I do not consider this a technical contribution.

3. I think Mosca is more suitable for and excels with monocular inputs. Therefore, I believe the authors have misclassified this 4D reconstruction method.

4. In my practice, Mosca performs very well on monocular videos, with excellent reconstruction quality and speed. Would the authors also report their metrics on the NVIDIA DyCheck dataset?

5. I agree with reviewers Jh53 and NHga that the evaluation is still relatively basic. I think the draft is not yet ready for publication in NeurIPS.

Additionally:
Based on my experience, Mosca reconstructs a scene in 30–40 minutes. Could the authors explain how they achieved a 5× faster training time?

Dear authors:

Thank you for the responses and efforts and appreciate the added comparative experiments with MOSCA which have addressed my concern.

1. I suggest the authors replace the "Pose-free" with "joint Pose optimization" and modify it throughout the paper, which is more in line with the scope of the method and helps avoid potential misleading.

2. From my perspective, Mosca belongs to monocular reconstruction methods and has a good performance, rather than being not comparable as initially claimed by the authors, which is confirmed by the authors' subsequent experiments, the performance improvement of 4D3R is limited.

3. I appreciate the authors providing additional extended experiments on Mosca (Extended Evaluation and NVIDIA DyCheck Results) and committing to include the comparative experiments with Mosca in the final paper. Additionally, I request the authors to report complete experimental metrics, which can be more convincing.

Literature Review: The mentioned works (CAT4D, 4K4DGen, GenxD, SV4D) have also achieved good reconstruction results using video input observations. A better paradigm might involve first generating denser observations, such as multi-view video or global world panoramas (e.g., HunyuanWorld), followed by 4D reconstruction. Therefore, I recommend that these works should be mentioned in the related literature even without direct comparison.

From my perspective, the original submission of the paper did have some technical issues: issues such as camera trajectories of the dataset, overclaim, weak evaluation, for example incorrectly categorizing COLMAP-based methods under Mosca and some data reports not quite matching my personal empirical values. I appreciate the authors addressing these issues through the rebuttal, so I have increased my score. But I suggest that the author conduct a detailed review of the paper, including technical details and metrics, and ensure the reproducibility of the method.

Dear Reviewer NHga,

We sincerely appreciate your thorough review and valuable insights. Your recognition of our MA-BA module and experimental results encourages us. We carefully address each concern and will implement your suggestions to strengthen our paper.

1. Dependence on Foundation Models

We acknowledge the reviewer’s concern about relying on pre-trained models. However, our findings indicate that this is a strength instead of a limitation:

• Leveraging proven technology: Foundation models such as DUSt3R and SAM2 represent state-of-the-art performance in their respective domains. Using them allows us to focus on the core challenge of pose-free dynamic reconstruction instead of reinventing components that already work well.
• Robustness analysis: We have conducted additional experiments on domain shift scenarios. When foundation models degrade (e.g., on out-of-distribution data), our method gracefully degrades to performance comparable to the MonST3R+SC-GS baseline while still maintaining functionality instead of complete failure.
• Clear Improvement: The ablation study (Table 3) shows that removing our motion-aware components causes 5.2dB PSNR drop. This demonstrates the value of our method beyond foundation models.
• Modular design: The foundation models in our framework can be replaced with newer versions or alternatives when they become available. This makes our framework modular and future-proof. We will add a detailed analysis of performance under foundation model degradation in the revised version.

2. Camera Intrinsics Assumptions

We clarify that our method estimates camera intrinsics as part of the optimization process (Eq. 4). The DUSt3R backbone provides initial estimates which are refined through our MA-BA module. For unknown intrinsics, we can incorporate standard calibration techniques or use approximate values with minimal impact on the performance of within 0.5 dB PSNR for ±10% focal length variation based on our additional experiments.

We will add a dedicated section discussing intrinsic parameter sensitivity and robustness in the revision.

3. Segmentation Quality Quantification

We agree this is important for validating our approach. We have computed the segmentation metrics on sequences where ground truth masks are available:

• On Sintel dataset (with motion annotations): IoU = 0.87±0.05
• On our synthetic test sequences: IoU = 0.91±0.03
• Compared to raw MonST3R masks: +0.23 IoU improvement

The visual examples in Fig. 3 are selected to show typical performance, they are not cherry-picked success cases. We will add comprehensive segmentation quality metrics and failure case analysis in the revision.

We greatly value the reviewer’s constructive feedback and commit to implementing all suggested improvements. The reviewer’s insights have helped us identify important areas for clarification that will significantly strengthen our contribution. We hope our responses address the reviewer’s concerns and demonstrate the robustness and practical value of our approach.

Dear Reviewer NHga,

The authors have provided responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Dear Authors,

I appreciate the authors providing a thorough response to my concerns. Most of my previous concerns have been addressed.

However, I agree with reviewer Jh53 and 8kQP that the evaluation dataset is limited. Unlike the authors' claim in their rebuttal, the cameras in DyNeRF dataset are static and do not contain any natural camera shake and drift. The authors should provide more results on widely used monocular 4D reconstruction datasets, such as NVIDIA, DAVIS and iPhone (DyCheck), for further comparison with current and future works.

Thank you for the authors’ efforts in conducting additional experiments on the NVIDIA and iPhone datasets. The numerical results show that the proposed method outperforms state-of-the-art method (i.e. MOSCA). Due to OpenReview’s limitations, I recommend also including additional visual figures for these experiments in the camera-ready version of the paper. I have no further concerns.