MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion

Q1-3: Clarification on the notation

1. The complete optimization in Eq. 6 minimizes the loss to obtain optimal values for X, P_W, and \sigma. However, it seems that the smoothness loss in Eq. 4 is minimized specifically for R and T. The input notation L_{smooth}(X) might be misleading—should it be L_{smooth}(R,T) instead?
2. The notation for L_{flow}(X) also seems ambiguous. Would L_{flow}(F) or L_{flow}(F,S) be more appropriate to reflect the actual inputs?
3. There is no explanation provided for the notation global. Clarifying what the global parameters are and how S^{global} is derived would be helpful. Specifically, is S^{global} in Eq. 5 an extension of Eq. 2 that uses F_{cam}^{global,t→t′} and F_{est}^{global,t→t′}

Thank you for your questions and suggestions. We have addressed these concerns as follows:

1./2.Regarding $L_{\text{smooth}}(X)$ and $L_{\text{flow}}(X)$: As discussed in L313, $\mathbf{X}$ is reparameterized by $\mathbf{D}$ (depth), $\mathbf{K}$ (intrinsics), and $\mathbf{P}$ (camera pose, including $\mathbf{R}$ for rotation and $\mathbf{T}$ for translation). Since $\mathbf{R}, \mathbf{T}$ in $L_{\text{smooth}}$ and $\mathbf{F}, \mathbf{S}$ in $L_{\text{flow}}$ can be derived from $\mathbf{X}$, we use $\mathbf{X}$ in Eq. 4–6 for brevity. This is also clarified in L316, where we state: “To simplify the notation for function parameters, we use $**X**^t$ as a shortcut for $\mathbf{P}^t, \mathbf{K}^t, \mathbf{D}^t$.” Additional clarifications have been added in Appendix Sec. B of the revised version.

3.Regarding the global notation: We have provided a detailed explanation of the global notation $F_{\text{cam}}^{\text{global}}$, in L333–L336 of the original paper. For the precise formulations of $F_{\text{cam}}^{\text{global}}$ and $S^{\text{global}}$, please refer to Appendix Sec. B.2, where we present these concepts in detail.

---

Q4: The proposed network is trained on diverse datasets, including PointOdyssey, TartanAir, Spring, and Waymo Perception, but the authors rely on a pretrained optical flow model. This could introduce a domain gap between the depth network and the optical flow network. Were there any issues related to domain gaps during the experiments?

Thank you for raising this point. While domain gaps can pose challenges, we observed that the pretrained optical flow model (SEA-RAFT) generalizes well across the datasets used in our experiments. This robustness is likely due to the nature of optical flow as a low-level task that relies on feature matching rather than high-level semantic understanding (e.g. the model can match similar textures). Consequently, optical flow models tend to generalize better to unseen domains.

In practice, we did not observe any domain gap for the optical flow model. It appeared to perform well across datasets. It is also worth noting that both our method and SEA-RAFT were trained on many of the same datasets, including TartanAir and Spring, and very similar style datasets KITTI vs Waymo.

Q: Camera Pose Estimation suite could be more comprehensive

1. E.g. COLMAP or masked COLMAP baseline would be good benchmarks.
2. Why are DROID-SLAM, DPVO and ParticleSfM not run on TUM, and DROID-SLAM and DPVO not on ScanNet?
3. LEAP-VO is highly competitive with the proposed method. This is a minor weakness given this method also estimates depth and improves on DUSt3R, but should be considered when analyzing e.g. if this paper should be considered for strong accept.

Thank you for the detailed feedback. Below, we address each point raised:

1. COLMAP and Masked COLMAP Benchmarks

   It has been reported in the ParticleSfM paper that (masked) COLMAP fails on 5/14 sequences on Sintel and 3/20 sequences on ScanNet. To provide a fair comparison, we evaluate MonST3R against (masked) COLMAP on the subset of sequences where COLMAP does not fail. As shown in the table below, MonST3R achieves superior performance across all pose evaluation metrics.

   Method	ATE ↓	RPE trans ↓	RPE rot ↓		ATE ↓	RPE trans ↓	RPE rot ↓
   	Sintel				ScanNet (20 seq)
   COLMAP	0.145	0.035	0.550		0.143	0.064	1.384
   COLMAP+MAT [A]	0.069	0.024	0.726		-	-	-
   COLMAP+Mask-RCNN [B]	0.109	0.039	0.605		-	-	-
   MonST3R	0.041	0.017	0.312		0.068	0.018	0.614

   Table 1: Pose evaluation metrics on Sintel and ScanNet (subset of sequences where COLMAP does not fail). We do not compare with masked COLMAP in ScanNet because it is of static scenes.

   [A] Motion-attentive transition for zero-shot video object segmentation, Zhou et al., AAAI 2020.

   [B] Mask R-CNN, He et al., ICCV 2017.

2. Comparison with DROID-SLAM, DPVO, and ParticleSfM on TUM and ScanNet

   The LEAP-VO paper compares extensively against DROID-SLAM, DPVO, and ParticleSfM and shows that LEAP-VO outperforms these methods. We find that MonST3R, despite tackling a more difficult task, performs comparable to LEAP-VO, a state-of-the-art method for pose-only prediction. We plan to include more comprehensive comparisons with these methods on TUM and ScanNet in future version.

3. LEAP-VO Comparison and Intrinsic Sensitivity

   We would like to highlight that LEAP-VO requires ground-truth (GT) intrinsics as input. To analyze its sensitivity to intrinsics, we conducted experiments with approximated intrinsics, for example, focal length derived from image width, which is a common practice in the literature [C, D, E], or using the focal length predicted by MonST3R. The results show a noticeable performance drop for LEAP-VO when using noisy or estimated intrinsics.

   Method	ATE ↓	RPE trans ↓	RPE rot ↓		ATE ↓	RPE trans ↓	RPE rot ↓
   	Sintel				ScanNet
   LEAP-VO w/ GT focal	0.089	0.066	1.250		0.070	0.018	0.535
   LEAP-VO w/ width focal	0.136	0.068	1.344		0.091	0.021	0.581
   LEAP-VO w/ MonST3R focal	0.125	0.063	1.280		0.075	0.020	0.573
   MonST3R	0.108	0.042	0.732		0.068	0.017	0.545

   Table 2: Pose evaluation on Sintel and ScanNet datasets. Best results without using GT focal are highlighted .

   [C] 3D Photography using Context-aware Layered Depth Inpainting, Shih et al., CVPR 2020.

   [D] 3D Moments from Near-Duplicate Photos, Wang et al., CVPR 2022.

   [E] SpatialTracker: Tracking Any 2D Pixels in 3D Space, Xiao et al., CVPR 2024.

Q1: Mention of Related Work: For video depth estimation, self-supervised monocular depth methods such as Manydepth [a] and SC-DepthV3 [b] should be mentioned. SC-DepthV3, in particular, provides video depth estimation results on the Bonn dataset, which would make for a relevant comparison.

Thank you for the suggestion. We have discussed these papers in the revised Related Work section (Line 175).

Additionally, we compared MonST3R with SC-DepthV3 on the Bonn dataset. As shown below, MonST3R outperforms SC-DepthV3:

Table: Video depth evaluation on Bonn dataset.

---

Q2: Robustness in Dynamic Regions: The statement “For most scenes, where most pixels are static, random samples of points will place more emphasis on the static elements” is generally true, but there are edge cases that require more robust approaches. For example, in the Bonn dataset, moving objects such as humans can occupy over 50% of the image. Potential solutions could include masking out dynamic regions using segmentation masks or re-running relative pose estimation after identifying dynamic regions.

Thank you for the insightful question. We have identified an issue in the explanation of our relative pose estimation method in the paper and have fixed it in the revision (Lines 251–260). Unlike traditional approaches, which rely on cross-view correspondences and thus dynamic regions will violate the static assumption, our method uses same-view correspondences, making it invariant to whether a region is static or dynamic.

• Relative Pose Estimation: Traditional methods often rely on corresponding points across two-views, using techniques like the epipolar matrix or Procrustes alignment to recover the camera pose. These approaches are sensitive to dynamic regions as they violate the static assumption inherent to these methods. In contrast, our approach leverages the fact that the output pointmaps of the second image by MonST3R model is expressed in the camera coordinate system of the first frame. Since each point is directly aligned with its corresponding pixel, we can use the Perspective-of-N-Points method to recover the relative pose. To further improve robustness, we employ RANSAC and define a valid mask by thresholding the estimated confidence mask to exclude low-confidence points.
• Global Pose Optimization: Our alignment loss $L_{\text{align}}$ (Eq. 3) leverages the correspondence between the global point cloud and the pairwise point cloud of the same frame, rather than using cross-frame correspondences. This design avoids the static assumptions required by cross-view correspondences, making the optimization process robust to dynamic regions.

We appreciate your suggestion regarding the use of segmentation masks to improve robustness and agree that adding additional information like semantics might be an interesting and promising future direction for improving these systems.

Q1: There are also some high-quality synthetic datasets in the video field, such as DynamicStereo and IRS Dataset. The author could consider expanding the scale of the training data to explore better model performance.

Thank you for the thoughtful suggestion to consider additional datasets such as DynamicStereo and IRS. We appreciate the recommendation and agree that exploring diverse datasets is a valuable avenue for future work.

Regarding these datasets:

• DynamicStereo: While it shares a similar distribution to PointOdyssey (mostly indoor synthetic data), which we already utilize extensively, we will consider evaluating whether it adds meaningful diversity or performance improvements in future work.
• IRS Dataset: This dataset focuses on static scenes and thus may have limited utility for training models aimed at dynamic scenes like MonST3R. Its coverage is also similar to TartanAir, which is already included in our dataset mixture.

Given the current scope and storage constraints, we prioritized datasets that align most closely with the objectives of this work. Expanding the dataset scale and diversity to further improve model performance is an exciting direction in the future.

---

Q2: There are doubts about the metrics used by the author when comparing video depth estimation. When the delta metric is below 0.7 or lower, it is usually considered that the prediction has completely collapsed. However, as far as the reviewer knows, the video depth estimation methods compared by the author have good performance and can provide basically correct depth estimation results at least on the Sintel dataset. Some metrics, such as NVDS, seem to be inconsistent with the report in the paper.

Thank you for the valuable comment. We would like to clarify that our evaluation protocol closely follows that of DepthCrafter. The performance of the baseline methods in our results aligns with those reported in the DepthCrafter paper.

It is important to note that we use per-sequence alignment, as opposed to the per-image alignment used by some other works, including NVDS (as evidenced in their official implementation). Per-sequence alignment is a more challenging evaluation setting, as discussed and demonstrated in the DepthCrafter paper. This difference in alignment protocol may explain why the reproduced NVDS results appear worse than the originally reported numbers.

Below, we compare NVDS and MonST3R under both per-image and per-sequence alignment protocols on the Sintel dataset. As shown, per-sequence alignment is more challenging than per-image alignment, and MonST3R consistently outperforms NVDS under both settings.

Table 1: Video depth evaluation on Sintel under different alignment protocols.

[A] Neural Video Depth Stabilizer, Wang et al., ICCV 2023.

---

Q3: It is necessary to explore the robustness of DUSt3R, that is, the failure cases when the scene is too dynamic.

We have included qualitative results highlighting DUSt3R's failure cases in dynamic scenes in Figure 2 and Figure A6 (rows 2-3) of the original paper. Specifically, DUSt3R struggles to align image pairs correctly when the scene contains significant dynamic motion, as it often misaligns based on the foreground objects or puts the dynamic regions into the background. In contrast, MonST3R successfully aligns the images based on the static region.

Quantitatively, the table below demonstrates DUSt3R's significant drop in performance on dynamic scenes, while MonST3R achieves a more robust performance.

Table 2: Video depth evaluation results on Sintel dataset.

Q1: The loss used in fine-tuning is not detailed in the methods section. In line 294, "static mask is both a potential output and will be used in the later global pose optimization", the word "potential" here is confusing, is the static mask also used during finetuning?

We use the same confidence-aware regression loss as DUSt3R for fine-tuning the model. We have updated the paper to include this information in Line 244. The static mask is not used during finetuning, but it is one of the downstream applications derived from the pairwise pointmaps output from the model. The static mask is also used for the global alignment process, as mentioned in Line 338 and Eq. (5).

---

Q2: Although a sliding window technique is used, the runtime reported is 30s for pairwise inference and 1min for global optimization given the 60 frames (around 2.5s) video input, which is slow.

Compared to previous methods in the same category, MonST3R is significantly faster. For instance, CasualSAM requires approximately 60 minutes and RCVD requires around 30 minutes to process the same 60-frame sequence on the same hardware. Additionally, MonST3R's runtime is comparable to depth-only methods, e.g., DepthCrafter, which takes about 3 minutes to handle 60 frames of video.

Currently, the majority of the run time is spent on global optimization and the intensive sampling of frame pairs. One interesting ablation is trying to eliminate these by running pointmap predictions between all the frames and an anchor frame. Then the global optimization would be unnecessary since all of the pairwise predictions would be in the camera coordinate frame of the anchor frame, and we can directly use the pairwise predictions as output. It also avoids extensive pair sampling since we only need to construct a single pair for each frame. Though this simplified approach has several limitations, such as sensitivity to the choice of the anchor frame and the lack of global information sharing across frames, it achieves real-time reconstruction at approximately 45 FPS on a single RTX4090 GPU and offers a promising direction for fully feed-forward reconstruction from monocular video.

We have included detailed explanations and additional examples in Appendix B of our revised version. Additionally, we have uploaded video examples of the real-time reconstruction results on our anonymous website. While not included in our original submission due to its limitations, we believe this mode serves as an effective ablation study to highlight the efficiency of MonST3R.

---

Q3: In the depth comparison in figure A1, the depth colormaps effectively hides the error, it's better to also add error colormaps. Additionally, in all the visualized frames, the model all predicts closer depth compared to ground truth on the dynamic parts. Is this specific to this dataset or is it a general bias of the model? What could be the possible reasons?

Thank you for the valuable feedback. We have updated our revision to include error maps alongside the depth maps (Figure A.1) for better clarity and to address your concern regarding potential visualization bias.

Regarding the observation that "the model predicts closer depth compared to ground truth on the dynamic parts," we believe this discrepancy arises from the visualization process. Specifically, min/max normalization was applied individually to each depth map in our previous version, and the presence of invalid pixels in the ground truth (GT) depth map can significantly bias the normalization, leading to an apparent discrepancy. In the revised version, we fixed it by using the same normalization scale for both the GT and all predicted depth maps. Based on the updated error maps and depth maps, it is clear that the foreground predictions are well-aligned with the ground truth.