Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding

Responses to Reviewer XuWq

Q1: The method is complex and hard to reproduce.

A1: Our design is intentionally modular to ensure simplicity and ease of replication. Each component (SAM2, Track Any Point, Depth Anything, and 4DGS) is an independent and standalone module, which makes the overall pipeline straightforward to reproduce. Our primary contribution is the unified optimization framework that connects these modules. This framework is conceptually simple, effective, and clearly detailed in Sections 3.2–3.4. We have also indicated in Line 20 of the paper that we will release the complete codebase including training and evaluation scripts upon acceptance to ensure full reproducibility.

Q2: Robustness and generalization of the method.

A2: The modular design of our method enhances robustness. Large pre-trained networks such as SAM2 and Track Any Point are known for their strong generalization across diverse datasets. By using these models, we improve the robustness of the 4DGS-based scene representation. As shown in Tables 1 and 2, previous 3D representation methods often fail to generalize to casual dynamic scenes in the DAVIS and DyCheck datasets, where uncontrolled motion and noise violate their assumptions. Although 2D networks generalize well in many scenarios, Figures 1 and 4 demonstrate their limitations under severe occlusions and complex dynamics. In such cases, 3D reasoning is essential to achieve consistent and reliable results.

Q3: Figure 2 is unclear.

A3: We appreciate the reviewer’s feedback regarding Figure 2. In the revised version, we will improve its readability by adding annotations to key components and refining the caption to better explain the pipeline.

Q4: Time consumption analysis.

A4: Our method differs from single-task end-to-end 2D models by building a full 4D representation that jointly produces segmentation masks, tracking, depth and rendered images for both training and novel views. This richer capability inherently requires per-scene optimization, as we model the scene’s 3D geometry and dynamics jointly rather than processing frames independently. Despite the joint optimization, our runtime remains comparable to other 3D-based pipelines. Optimizing a scene with 100 frames typically takes about 30 mins on a single 4090 GPU, which is close to the 4DGS baseline (~24mins). After optimization, the rendering is highly efficient, at about 75 fps, which enables real-time rendering and querying for segmentation and tracking results. We will clarify by adding a detailed runtime analysis in our final paper.

Q5: Table 3 is unclear.

A5: We thank the reviewer for pointing this out. There is indeed a typo in Table 3, which we will correct.

Q6: Replacing the confidence $w(p)$ with a hand-crafted value.

A6: The confidence $w(p)$ is an adaptively learned logit, with its ground-truth defined based on the rendered result. This is similar to TAP model designs, where the predicted confidence logit is supervised by a binary signal, which is set to 1 when the prediction is within a predefined threshold and 0 otherwise. This binary signal serves only as a training target and enables the model to learn a smooth confidence field in 3D space. In contrast, hand-crafted confidence measures often yield brittle and noisy weights, particularly in regions with similar textural structures.

Q7: Comparison with SA3D.

A7: Our iterative pipeline shares a similar prompting mechanism with SA3D, but differs substantially in design and scope:

• (a) Target setting: While SA3D is designed for static scenes and focuses solely on propagating a single-view mask to multi-view masks, our method targets monocular videos in dynamic scenes and refines SAM masks across the entire sequence to construct a coherent semantic representation.

• (b) Temporal consistency: In addition to multi-view consistency, our method explicitly enforces temporal consistency by combining SAM prompting with sequential optimization of semantics as shown in Figure 3(b). This enables masks to propagate smoothly across frames and align segmentation with dynamic motion.

• (c) Iterative refinement: Our approach jointly refines both the 2D priors and the 3D semantic field, instead of only updating a semantic voxel grid as in SA3D. This design allows us to progressively improve mask quality across frames and views by re-prompting SAM using the corrected inputs. In comparison, SA3D queries SAM only once per frame with no feedback loop.

We will include this discussion and compare our method with SA3D in the final paper.

Responses to Reviewer 8CeR

Q1: 2D prior dependency.

A1: Although our method uses multiple 2D priors, our objective is to refine and correct these priors through 3D consistency and temporal coherency. As demonstrated by the results in Figure 1 and 3, our iterative optimization substantially improves the input 2D priors even when they are noisy. Our design offers a distinct advantage over both pure 2D and 3D methods. Pure 2D networks often produce inconsistent outputs, particularly in regions with severe occlusions and motion ambiguity. Furthermore, 3D baselines struggle to leverage priors as effectively as our approach. Consequently, our refinement process is essential to achieve stable and robust results despite inheriting some limitations of the 2D priors.

Q2: Improvement on DAVIS 2017 val is trivial.

A2: The modest gain on DAVIS 2017 val arises because the baseline (SAM2) already achieves near-saturated performance (over 90%) on this split. While diverse in content, the DAVIS dataset contains short video clips (fewer than 100 frames) with salient foreground targets and limited occlusion. In contrast, our method shows clear improvements on more challenging scenes (over 400 frames) in the DyCheck dataset (Tables 1 and 2), which require long-term tracking across extended sequences. Even on DAVIS val, our method enhances temporal consistency across frames (Figure 1), which is not fully captured by raw performance scores.

Q3: Missing ablation on optimization tasks.

A3: Tracking and depth cannot be treated as independent tasks as each track corresponds to a 3D trajectory. Since 3D tracks are not directly observable, we supervise the motion field using both 2D tracking and depth cues. Ablating either signal leads to a significant performance drop: using only tracking reduces $<\delta_{avg}$ on DyCheck from 50.4 to 41.6, and using only depth yields 25.9. This highlights that both signals are essential and complementary. Similarly, training solely on segmentation is not viable since the semantic field relies on the underlying 3D geometry. Without the motion field (tracking and depth), the model fails to converge and produces unusable results. We will include these additional ablations in the final paper to clarify the necessity of the full design.

Q4: The contribution of each component.

A4: The effectiveness of our adaptive resampling module has been shown in the ablation study in Table 3 (“w/o Adaptive Sampling”). An ablation for iterative semantic refinement, which is applied during the sequential optimization stage, has also been presented in Table 3 (“Full Initialization”). In addition, we show the ablation results without confidence map here, which achieves 89.4 $\mathcal{J}$ & $\mathcal{F}$ and 48.3 $<\delta_{avg}$. This shows that our confidence map is effective in improving the performance. We will make the contribution of each component more clearly in the final paper.

Q5: How does the adaptive resampling strategy handle regions that are under-represented due to occlusion?

A5: Our adaptive resampling strategy operates only on visible regions. Fully occluded areas cannot be directly reconstructed since their true geometry is inherently unobservable from the available data. Nevertheless, as shown by the tracking results in Figures 1 and 5, our 3D representation remains more stable in under-represented regions compared to the 2D networks. This suggests that our 3D representation is capable of producing plausible and coherent results by enforcing spatial and temporal consistency, even when direct observations are unavailable and 2D predictions are drifting.

Q6: Sequential-global optimization is resource-heavy and time-consuming.

A6: We acknowledge that achieving real-time performance during optimization is beyond our current scope. However, our primary goal is to produce accurate and temporally consistent results over long sequences, which is challenging for lightweight pre-trained 2D models. Despite the computational overhead, our runtime is comparable to other 3D-based pipelines: optimizing a scene with 100 frames typically takes approximately 30 minutes on a single 4090 GPU, close to the baseline of 4DGS (~24min). After optimization, the rendering is highly efficient, at about 75 fps, which enables real-time rendering and querying for segmentation and tracking results.

Responses to Reviewer D7AY

Q1: The overall concept of the paper is rather incremental.

A1: We thank the reviewer for the valuable feedback. While Motion4D leverages existing 2D and 3D components, its contribution goes beyond serving as an auxiliary refinement module. Motion4D constructs a joint 4D representation that integrates motion, depth, and semantic cues. This directly addresses the reviewer’s call for “a unified framework and joint optimization of multiple tasks”. Crucially, the joint optimization of 2D priors and an explicit 3D representation is not only complementary but fundamental to the effectiveness of our approach. We will revise the final paper to clarify and emphasize that Motion4D is a carefully designed joint optimization framework rather than a simple refinement pipeline.

Q2: Visualization of ablation settings.

A2: We appreciate the suggestion to include additional visualizations and will incorporate them in the final paper. Although figures cannot be attached in this rebuttal, we highlight the key findings from our ablation studies. Particularly, removal of the adaptive resampling module leads to unreliable reconstruction in fast-moving or occluded regions such as the legs of dynamic objects. Figure 3 of the supplementary material shows the importance of this component, where the baseline without resampling exhibits noticeable artifacts and even missing regions in the challenging regions. In addition, without global optimization, we observed that the Gaussians may be shifted abruptly between chunks, leading to clear motion drift.

Q3: Optimizing depth estimation.

A3: Depth is optimized jointly with 2d tracks as part of the iterative motion refinement process. Specifically, Equation (5) defines the depth loss as the discrepancy between the 3D-rendered depth and the 2D depth priors, formulated in a manner consistent with the loss for 2D tracks.

Q4: Comparison on 3D point tracking benchmarks.

A4: We have evaluated 3D point tracking performance on the DyCheck benchmark, which includes tracking annotations and LiDAR depth. The results are reported in Table 1 of the supplementary material. Our Motion4D demonstrates its ability to maintain accurate long-term 3D correspondences by achieving consistent improvements over existing 3D methods. We will emphasize these results and provide more detailed discussion in the final paper.

Q5: Evaluating the mIoU metric for segmentation results.

A5: Our evaluation follows the standard $\mathcal{J}$ & $\mathcal{F}$ metric widely adopted in video object segmentation. Here, the $\mathcal{J}$ score corresponds to the mean Intersection over Union (mIoU) across frames, and the $\mathcal{F}$ score measures contour accuracy. This means that our reported $\mathcal{J}$ & $\mathcal{F}$ results have included mIoU which ensures comparability with other VOS benchmarks.

Q6: Computational cost of the optimization process.

A6: Our model is trained and evaluated entirely on a single RTX 4090 GPU and contains approximately 150M parameters, depending on the scope of the scene. The training phase of our Motion4D is not real-time since it relies on predictions from multiple pre-trained networks and follows the per-scene optimization paradigm of 3DGS-based methods. Despite this, our runtime is comparable to other 3D-based pipelines: optimizing a scene with 100 frames typically takes approximately 30 minutes on a single 4090 GPU, close to the baseline of 4DGS (~24min). The rendering of segmentation and tracking results is fast, at about 75 fps, after optimization since the 4DGS representation can be rendered efficiently. We will clarify in the final paper that real-time performance during optimization is beyond our current scope. The primary goal of our paper is to deliver accurate and temporally consistent results over long sequences, which is challenging for lightweight pre-trained 2D models.

We appreciate the reviewer's suggestion to add new experiments on the TAPVid-3D benchmark, and we will include them in the final paper. However, due to limited time during the discussion period, we were only able to provide a experiment on the "minival" split of the DriveTrack and PStudio subsets:

Here, we report the 3D Average Jaccard (3D-AJ), the average percent of points within $\delta$ error (APD), and the occlusion accuracy (OA). As shown in the table, our method outperforms SpatialTracker across all metrics for 3D tracking performance. We believe this improvement stems from our explicit 3D representation, which allows the method to leverage spatial geometry more effectively than pure 2D networks.

Responses to Reviewer iUVL

Q1: Lack of visualization of confidence and uncertainty maps.

A1: We will include these visualizations in the revised version, but cannot provide them here due to the rebuttal’s format constraints. The "uncertainty logits" refer to the learnable attributes attached to the 3D Gaussians (one uncertainty logit per 3D Gaussian). We use this 3D field (the learnable attributes) to render 2D views of the confidence maps, which is similar to how 3D Gaussian colors are rendered into images. This approach allows us to optimize a 3D-consistent uncertainty field that reflects the underlying scene geometry, and use the rendered confidence maps to update 2D priors.

Q2: Lack of visualization of the optimization progress.

A2: We will include this visualization in the final paper. Nonetheless, we note that Figure 3 has provided a related illustration of the optimization progress. The figure shows our Motion4D progressively refines the inconsistent 2D priors and produces temporally consistent results during the sequential optimization stage.

Q3: The deformation field is a bit confusing.

A3: The deformation field follows the formulation introduced in prior work [1]. Instead of defining $N$ bases, where $N$ is the number of Gaussians, we define $B$ global bases ($B \ll N$), where each base contains $T$ elements in $\mathbb{SE}(3)$. We assign a coefficient vector of length $B$ for each 3D Gaussian, whose per-frame transformation is then computed as a weighted sum of the $B$ elements in $\mathbb{SE}(3)$ at that time step. This design significantly reduces the number of parameters and enables smooth temporal deformation. Although we adopt the same deformation field formulation, we extend it to integrate with the adaptive resampling module in Motion4D. This is why we kept its description concise in the method section. We also evaluated a variant using per-Gaussian transformation. While this increases flexibility, it results in slightly worse performance ($<\delta_{avg}$ drops from 50.4 to 49.6 on DyCheck) due to overfitting and unstable tracking estimation. In the final paper, we will expand the explanation and add this ablation to clarify its effect.

Q4: The proposed evaluation is quite small.

A4: The proposed DyCheck-VOS benchmark consists of long video sequences, each with 200–500 frames. For comparison, the DAVIS 2017 validation set for segmentation and tracking includes 30 short videos, each with fewer than 100 frames. As a result, DyCheck-VOS introduces greater challenges in long-term tracking, especially in handling frequent occlusions, spatial alignment, and object re-identification over extended sequences. Due to the difficulty of annotating dense point trajectories for Track-Any-Point, a larger benchmark is currently unavailable for evaluation. We will include additional examples to better showcase the complexity and challenges posed by Dycheck-VOS.

Q5: Failure cases on DyCheck-VOS.

A5: Our method may fail in scenarios where the underlying 3D reconstruction is unreliable, typically due to inaccurate SfM camera poses or extremely challenging scene geometry. Nonetheless, our proposed iterative refinement framework can address several failure cases when the 2D priors are noisy and unreliable. These cases are shown in Figure 3, where the input segmentation and tracking results are poorly estimated. In the final paper, we will elaborate on the causes of such failure cases and discuss potential strategies for mitigating them in future work.

[1] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407.13764, 2024.