MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details

We appreciate the reviewer’s valuable feedback and insightful questions.

1. Regarding the choice of radius $r$ in mismatch filtering.

   Our strategy follows the same multi-scale radius selection as in MoGe's local loss, which has been verified as effective for leveraging context at different spatial scales. Specifically, we select three radii such that their projected image regions approximately cover 1/4, 1/16, and 1/64 of the image size, respectively. This multi-scale design allows the filtering to capture and suppress misalignments of varying scales using both coarse and fine local evidence. We will include this explanation and the exact computation formula in the revised version for completeness.

2. Clarification on global shift in training.

   We assume the reviewer is referring to the training-time global shift used in supervision. As stated in Line 154–155, "we maintain the geometry branch for affine-invariant point map as in MoGe, ..." We are happy to elaborate further for clarity.

   In both the main model and all scale-decoupled ablations, the affine point map branch follows exactly the same ROE alignment strategy as MoGe in training supervision. Specifically, the predicted affine point map is aligned to the ground truth point map via ROE alignment, which jointly solves for a global scale and shift to minimize the ROE loss (i.e., L1 under best affine alignment).

   $$(s^*, \mathbf{t}^\*) = \arg\min_{s, \mathbf{t}} \sum_{i \in \mathcal{M}} w_i \Vert s \hat{\mathbf{p}}_i + \mathbf{t} - \mathbf{p}_i\Vert_1$$

   This alignment yields:

   • the optimal global scale $s^*$, which is also used as ground-truth for supervising the metric scale prediction branch as decribed in Eq.(3).

   • the optimal global shift $\mathbf t^*$.

   The metric scale prediction branch thus learns to regress the global scale $\hat s$ directly, with supervision from the ROE-derived $s^*$ as decribed in Eq.(3).

   Only in the special ablation, Entangled Shift-invariant, where the predicted point map is not affine-invariant but metric-scaled with unknown shift, the alignment is done with shift only:

   $$\mathbf{t}^* = \arg\min_{\mathbf{t}} \sum_{i \in \mathcal{M}} w_i \Vert\hat{\mathbf{p}}_i + \mathbf{t} - \mathbf{p}_i\Vert_1$$

   We will make this distinction more explicit in the final version and include the relevant formulation for completeness.

3. Clarification on "absolute bias".

   We appreciate the reviewer’s attention to this point. The term absolute bias refers to the systematic deviation of geometry predictions caused by the lack of real-scene prior knowledge when training purely on synthetic data. While synthetic-trained models can often recover local relative geometry accurately, they may produce biased estimation on scene-level geometry and layout when applied to real-world images.

   This is especially evident in large-scale indoor or outdoor scenes, where synthetic models are unfamiliar with real-scene object proportions or layouts. As illustrated in Figure 5, the model trained on synthetic data reconstructs the relative shape of the chair well, but places it too close and too small compared to the ground truth, revealing a global misinterpretation of the scene layout. We use the term absolute bias to describe such behavior, where the local structure appears plausible, but the global layout is misaligned.

   This observation motivates our choice of a local filtering strategy (rather than relying on global consistency), which is more robust to these domain-induced absolute errors. We will clarify this term further in the revision to avoid ambiguity.

4. Failure cases

   We agree that visualizing failure cases can provide useful insights. As noted in our paper, failure typically arises in scenes with extremely fine structures, significant foreground-background scale disparity, or strong out-of-distribution ambiguity in metric scale. We will add some visualizations of the failure cases in the revised version of our paper.

---

Please let us know if you have further questions.

We thank the reviewer for the detailed and constructive feedback, as well as the recognition of our motivation, design, and experimental results. We reply to the reviewer's concerns as follows.

1. Concerns about Novelty.

   We would like to clarify the novelty and key insights behind our design:

   • Network design for scale prediction: While the final design of using a CLS-token MLP appears simple, it is the result of both a key insight and careful empirical validation. This insight is that metric scale is inherently a global property and is better predicted by decoupling from affine dense prediction and leveraging global feature. Importantly, this was not an arbitrary architectural choice. We experimentally compared several alternative designs and found that the CLS-token MLP yielded better results (Tab. 4). To the best of our knowledge, this specific strategy has not been explored or experimentally validated in prior works on metric depth or 3D point map recovery.
   • Real-world data refinement: To achieve both real-world geometry accuracy and sharp details, we propose a nontrivial refinement pipeline that addresses structured artifacts commonly present in real-world RGB-D data (e.g., SfM, LiDAR). Our empirical observations reveal a duality in such labels that they are partially accurate but often degraded by systematic noise. The key challenge and also motivation behind our design is to exploit the reliable regions while avoiding degradation from corrupted areas. Our method selectively filters out noisy regions and fills them in using a model trained on clean synthetic data. This pipeline is essential for bridging the synthetic-to-real domain gap, enabling robust and high-fidelity supervision across diverse real-world sources.

   In summary, our designs emphasize simplicity without compromising effectiveness. We believe this type of simple design with strong performance, supported by analysis and experiment, constitutes a meaningful contribution.

2. MLP v.s. Conv head.

   We thank the reviewer for the insightful suggestion. The proposed alternative, which repeats and concatenates the CLS-token feature to the Conv head input for metric scale prediction, indeed introduces global information and partially decouples the scale from structure. We implemented the reviewer's suggested design and found that it does improve performance over Conv head without CLS-token feature, confirming that global features are important for predicting scale. However, it remains less effective than directly applying an MLP to the CLS-token. As summarized in the table below, the MLP head demonstrates slightly better overall performance (wins in 11 out of 17 metrics, ties in 2), and remains the most straightforward and effective design while being simple and lightweight in practice.

   Configuration	|	Metric			|					Relative							|Sharpness
   	|Point		|Depth		|Point						|Depth						|
   					|Scale-inv.		Affine-inv.		Local		|Scale-inv.		Affine-inv.		Affine-inv.(disp)		|
   	|Rel↓	δ₁↑	Rel↓	δ₁↑	|Rel↓	δ₁↑	Rel↓	δ₁↑	Rel↓	δ₁↑	|Rel↓	δ₁↑	Rel↓	δ₁↑	Rel↓	δ₁↑	|F1
   Conv head	|9.62	91.4	17.6	68.4	|12.2	86.3	9.15	90.0	6.34	94.9	|8.46	90.2	6.62	93.2	7.74	92.1	|12.7
   CLS token + Conv head	|9.02	92.0	16.7	69.7	|11.9	87.6	8.93	90.5	6.21	95.1	|8.36	90.6	6.57	93.5	7.62	92.3	|12.3
   CLS token + MLP head	|9.20	91.9	16.5	72.8	|11.6	87.6	8.87	90.6	6.26	95.1	|8.23	91.0	6.53	93.4	7.53	92.6	|12.5

3. Using of GT scale factor ($s^*$).

   We thank the reviewer for pointing out that the details of the ground-truth scale $s^\*$ were under-explained. As proposed in MoGe, the $s^*$ is the optimal scale solution under a scale-and-shift alignment with respect to a weighted least absolute deviation (LAD) problem:

   $$(s^\*, \mathbf{t}^\*) = \arg\min_{s, \mathbf{t}} \sum_{i \in \mathcal{M}} w_i \Vert s \hat{\mathbf{p}}_i + \mathbf{t} - \mathbf{p}_i\Vert_1$$

   The ROE solver divides this problem into a series of searching subproblems in parallel, utilizing the piece-wisely linear property of the objective landscape. We will include additional explanation to supplement the preliminary section.

   Regarding the choice of GT scale, we believe the use of the ROE-derived scale $s^\*$ as ground truth is well-motivated and principled. By definition, $s^\*$ minimizes exact ROE loss to best match $s \hat{\mathbf{P}}$ with $\mathbf{P}$, and is thus a natural supervisory signal aligning with the training strategy of the affine-invariant ROE loss in MoGe to inherit its superior performance.

   To address the reviewer’s suggestion of using a simpler closed-form target (e.g., mean distance from centroid), we implemented a variant using median-centered normalization and scale defined by average distance—similar to the approach in MiDaS. However, this alternative led to degraded performance on metric prediction:

   • Metric depth AbsRel error increased from 16.5% → 17.8%,
   • Metric point AbsRel error increased from 9.20% → 10.2%.

   These results confirm that using ROE-aligned $s^*$ yields better supervision under our task formulation.

4. Depth filtering / completion on large missing areas.

   Our refinement pipeline remains effective even when large regions are filtered out. Although the inpainted content may not be perfectly accurate, the Poisson inpainting preserves structural sharpness and edge alignment due to its gradient-based formulation. Crucially, the filled regions are at worst comparable in quality to the original synthetic predictions, ensuring that supervision quality is not degraded. In contrast, leaving these regions empty would create large unsupervised areas during training, which could harm model performance in those regions.

---

Please let us know if you have any further questions.

On the novelty of scale prediction.

---

We appreciate the reviewer’s feedback and the insightful references.

Clarification about the references

1. "The design for metric scale estimation shown in Figure 3(a) shares similarities with existing approaches such as ZoeDepth[1] and UniDepth[2]"

   Figure 3 has three subfigures, 3-1, 3-2(a) and 3-2(b). We assume the reviewer is referring to either 3-1 or 3-2(a) (please let us know otherwise)

   We would like to clarify that both Figure 3-1 ("Entangled") and Figure 3-2(a) ("Decoupled, Conv head"), which the reviewer associates with ZoeDepth and UniDepth, are in fact part of our ablation baselines for the purpose of validating our final design in Figure 3-2(b).

   While these baseline settings may resemble prior works, they are not the MoGe-2 architecture, but part of our contribution to provide controlled comparisons and demonstrate the effectiveness of our decoupled metric scale estimation design.

   • UniDepth separates camera and depth modules, but does not decouple global metric scale from depth. In fact, our ablation experiment "Entangled + SI-log Loss" is designed to replicate UniDepth's formulation and learning objectives (SI-log for depth, MSE for camera UV map) to offer a fair and controlled baseline comparison.

   • ZoeDepth adopts an AdaBins-style formulation with per-pixel bin classification for both depth and metric modules. While ZoeDepth includes a scale estimation component, its formulation and objective are different from our simpler direct regression design.

2. "for Figure 3(c), the use of the CLS token to capture global image-level information for scale estimation is conceptually aligned with prior works like ScaleDepth[3] and RSA[4]"

   We assume the reviewer is referring "Figure 3(c)" to the 3rd subfigure, Figure 3-2(b).

   Regarding ScaleDepth and RSA, while we acknowledge their use of global features to resolve metric scale, they both rely on text input and language models (e.g. CLIP) to provide such semantic feature. In contrast, our method explores how to leverage purely image-based features from visual-only pretraining (e.g., DINOv2), tailored for metric point map estimation. Our approach tackles a different and much simpler setting, and is also evaluated as effective by experiments.

Clarification of the novelty of our scale prediction

To our knowledge, no prior work has proposed or systematically validated directly predicting the global scale from CLS tokens in this context, with exclusive supervision and loss alignment as we propose. Behind the simple yet effective final design is the result of both a key insight of decoupling relative geometry and metric scale and careful empirical validations. We believe this type of simple design with strong performance, supported by analysis and experiment, constitutes a meaningful contribution.

 

On novelty of leveraging real-world RGB-D data

---

Differences from Depth Anything V2

" This design is similar to the training strategy of DepthAnything-v2... The main difference lies in using RGB-D rather than RGB data for pseudo-label generation."

We wish to point out that the goals and mechanisms of Depth Anything V2's pseudo labeling and our real-world data pipeline differ fundamentally. They serve orthogonal purposes as summarized below:

Depth Anything v2 focuses on distillation using the pseudo labels generated by a stronger teacher model, where the unlabeled diverse RGB images serve as a bridge for knowledge transfer. In contrast, we do incorporate real RGB-D ground truth labels, and propose a new filtering and refinement pipeline to address its artifacts to improve metric geometry accuracy without sacrificing sharpness.

Using a larger teacher model for pseudo-label distillation is orthogonal to our real-world data refinement pipeline. In fact, the two strategies could be used on top of each other, e.g., further distilling from a ViT-Giant MoGe-2 trained on refined real data.

To isolate the impact of pseudo-labeling itself—rather than gains from scaling model size, we followed the reviewer’s suggestion and replaced our filtered real GT depth with pseudo-labels from a model of the same capacity trained with synthetic data in ablation studies (ViT-B). As shown in the table below, the resulting performance is close to using synthetic-only training, and significantly inferior than using real labels.

These results indicate that pseudo labels alone do not provide free gains from a same-capacity model, and highlight the value of leveraging refined real RGB-D label for metric supervision.

The necessity and effectiveness of real-world RGB-D data

"The diversity and quality of synthetic datasets are not inherently inferior to those of real-world RGB-D datasets"

We respectfully disagree with the reviewer's assertion that (currently available) synthetic data alone can sufficiently cover the real-world RGB-D distributions.

As clearly shown in Table 4 in the paper, compared to those trained with real data, models trained solely on synthetic data suffer from a notable drop in accuracy (e.g. metric depth AbsRel increases 15.8%→21.7%, affine-invariant disparity 7.53%→8.37%) on all metrics except for sharpness. This reflects the known limitations of synthetic data: Domain gap and limited scene coverage (also recognized in Depth Anything V2's Section 3, but they choose to use giant model distillation to improve large and base model's performance).

Our work builds on this understanding, and provides a principled and effective pipeline to incorporate real RGB-D data while mitigating its artifacts to avoid degrading sharpness. This is a core motivation backed by both existing literature and our ablation results.

---

Please let us know if you have further questions.

We sincerely thank the reviewer for the constructive and encouraging feedback. Your clear summary and thoughtful questions help us better articulate the implications and potential extensions of our work.

1. DINOv2 backbone finetuning; Scale correlation with scene types.

   All model parameters, including the DINOv2 backbone, are trainable during training. We follow the same initialization and optimizer setup as MoGe: DINOv2 pretrained weights for the backbone, and randomly initialized heads, trained end-to-end (see lines 219–225 in Implementation Details).

   The reviewer raises an insightful point regarding the potential correlation between the CLS token and scene types. Indeed, the CLS token of DINOv2 encodes strong global semantics. For example, according to DINOv2's official benchmarks, the frozen ViT-Large backbone's CLS token achieves 83.5% (k-NN) and 86.3% (linear probe) top-1 accuracy on ImageNet classification, demonstrating its strong capacity to represent scene and object semantics. We believe this capacity to implicitly recognize scene types and iconic objects with known sizes is directly relevant to metric scale estimation.

   As suggested by the reviewer, we analyzed the predicted global scale factor from the MLP across two datasets:

   • Indoor (NYUv2): Mean = 1.79, Std = 0.44

   • Street-view (KITTI): Mean = 11.9, Std = 2.0

   This clustering suggests that our MLP head over the CLS token indeed captures meaningful scene-type-aware priors for metric scale estimation.

2. Including more examples of recognizable landmarks

   We appreciate the reviewer’s thoughtful suggestion. While the rebuttal format limits us from including more examples, we clarify that the scene in Figure 6 (3rd row) comes from a real-world photo of 135 East 57th Street (Tower Fifty Seven) in Manhattan, New York. This building has a known height of 131.1 m, and our model estimates it as 116 m. In the same image, a New York yellow taxi (Ford Crown Victoria, approximately 1.45 m tall) is also present. Our model predicts it as 1.49 m. These consistent predictions across both large-scale and small-scale objects suggest that our model can effectively ground metric estimates using recognizable reference cues in the scene.

   We agree that using landmarks with known heights is a valuable way to validate scale estimation accuracy. At the same time, we note that certain landmarks with highly distinctive and atypical structures may present challenges for our model, due to the lack of familiar contextual cues. For example, when tested on images of the Eiffel Tower, the model estimates a height of 70–100 m, which may appear reasonable in isolation but underestimates the true height (330 m). Interestingly, on the Las Vegas replica, which shares a similar appearance but stands at roughly half the height (165 m), the model predicts a similar range of 70–120 m, which is closer to the true height. These cases highlight potential areas for future improvement and underscore the importance of visual context in scale reasoning. We will release an open demo to help users explore and better understand such generalization behaviors.

---

Please let us know if you have further questions.