Evaluating Robustness of Monocular Depth Estimation with Procedural Scene Perturbations

Thanks for your constructive feedback, and we are glad that you found the importance of introducing robustness to monocular depth estimation (MDE) assessment and that our findings could guide future improvements. We have addressed the individual concerns below.

Concern 1 (and question 1): Results on synthetic data may not generalize to the real world.

While we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world.

In addition, our synthetic images are legible to humans in terms of their 3D structure. The fact that models struggle on them reveals potential misalignment of models with human perception, which can help guide model development.

To address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We then captured 15-20 images per variation and manually annotated the pose and object mask to infer a precise depth map for the object. We present the results below. Since this is a limited experiment in the context of objects on a table top (similar to the domain of robotic arms), one should not expect the results to perfectly match our much more diverse synthetic evaluation.

Error

Stability

We find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.

Concern 2 (and question 2): Only one perturbation is evaluated at the time, while real-world images often involve multiple factors combined. It would be valuable to see the results for combinations of perturbations

We believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.

Some of the perturbations (in-distribution perturbations) already exist in the base scenes, e.g., diverse lighting conditions, materials, and camera tilt angles; some of them (out-of-distribution perturbations) do not, e.g., camera roll angles are typically standardized in the base scene. For those in-distribution perturbations, we expect that combining perturbations will have at most a small impact on the average error, while the instability may increase due to the greater difference between the perturbed scene and the original. When combining out-of-distribution perturbations, we would expect the average error to increase, though the exact effect is unclear.

We present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks.

This confirms our expectations about the interactions between variations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.

Concern 3: Confusing labeling of Table 3

We will update this to read Self-(in)consistency.

Question 3: I’m interested to see results of VGGT [1] – does multi-image training lead to greater robustness to camera perturbations?

We present results for VGGT below.

VGGT is not competitive with top models. For average of all three metrics over perturbation types, VGGT is worse than Depth Pro and UniDepthV2) The poor average error might be due to VGGT outputs depth maps at a low resolution (294x518). To perform evaluation, we upsample the depth maps using nearest neighbor interpolation. Regarding metrics that evaluate robustness to camera-related perturbations (Accuracy (in)stability and Self-(in)consistency on Cam Dolly Zoom, Cam Roll, Cam Pan/Tilt), it is also not the best. This may suggest that multi-image training may not directly lead to greater robustness to camera-related perturbations. (Note that low operating resolution does not affect robustness much, as predictions for base is also made under low operating resolution).

[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5294-5306.

Thanks for your constructive feedback, and we are glad that you found our paper well-written and that our benchmark is valuable in understanding the robustness of models to various perturbations. We have addressed the individual concerns below.

Concern 1: It is unclear how we guarantee that the object of interest is sufficiently salient in the scene given that the fish in scene 3674e176 is fairly small.

The particular fish mentioned is the third smallest of all objects, has an object mask with 3970 pixels, and has a bounding box size of 7685 pixels; it would be classified as medium size by COCO’s object categories. To ensure that objects are sufficiently salient we generate scenes with Infinigen and then manually select specific objects and scenes. We sought to encompass a reasonable distribution of object placement – we didn’t want all objects to be large and perfectly centered in the frame, but we also didn't want them to be occluded by other objects. To get a better sense of the distribution, the median object has a bounding box size of 92308 px (~= $304^2$ px). Using COCO’s small/medium/large object categories (appropriately scaled to our image resolution) we find we have 0 small objects, 8 medium objects, and 32 large objects.

Concern 2: Reference to camera pedestal/truck in section 3.1

Thanks for pointing this out. This variation is not present in the dataset and we will correct this section to remove the reference.

Concern 3: Labeling of Table 3 is confusing/incorrect.

We will update this to read Self-(in)consistency.

Concern 4: There are no evaluation results on the full scene

In figure 2 (L212) we display the average AbsRel error on the full scene. We elected to focus on object-level comparisons since these are much more precise and avoid the ambiguity present in full scene MDE. We still do agree that there are interesting insights to be gained from more closely analyzing robustness on the entire scene. We present a summary of the results below.

Among the variations, the instability largely aligns with the expectations based on the magnitude of the pixel-level change in the scene. For example, object material swap only affects the relatively small area of the object and so has very low instability. Notably, lighting also has low instability despite impacting larger portions of the scene. Among the different models, we observe that MoGe achieves the best performance by average error while UniDepth has the best accuracy (in)stability. We will include complete tables and more thorough analysis in the appendix.

Question 1: What is the motivation for some perturbations? Why is the 2D size of objects roughly maintained? Is it to keep the saliency and the visibility?

For each perturbation, we sought to make minimal alterations to the depth map of the object so that the task complexity is unchanged and performance on the perturbed scenes can be directly compared. Saliency and visibility are important components of this since there may be less visual information if the object is smaller. However zooming in on the object while maintaining its apparent size will preserve geometric information. Similarly in the case of object translation and rotation we only select perturbations in a “small neighborhood” so that the depth map is not dramatically changed by revealing another side of the object.

Question 2: Would computing alignment using only the depth values for the object lead to metrics that capture only local robustness/consistency and potentially missing the global inconsistency? What if the alignment is computed on the entire scene and then the error metrics are computed on the object area?

Yes, alignment using only the depth values for the object misses global inconsistencies. As discussed above, we agree that global consistency is important and can be captured with metrics on the full scene.

However, we do not find it reasonable to compute the alignment on the entire scene and only evaluate metrics on the object area. We design our perturbations such that the complexity of the task of predicting the object geometry is unchanged, but the same does not hold for predicting the background geometry. For example, a camera pan/tilt or lighting variation may alter the background such that its depth becomes ambiguous. Then, computing alignment on the background image could introduce distortions which even a robust model would be unable to avoid. Moreover, it would be difficult to interpret the magnitude of the error and its implications for assessing the model.

Concern 1: Results on synthetic data may not generalize to the real world

While we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world.

To address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We captured 15-20 images per variation and manually annotated the object’s pose in order to compute a precise depth map. Since this experiment is limited to a single object on a table, one should not expect the results to perfectly match our much more diverse synthetic evaluation. We display selected models below and will include the full results in the paper.

Error

Stability

We find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.

Concern 2: Real-world scenarios often involve combined perturbations.

We believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.

We present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks.

These results align with our expectations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.

Concern 3 (+ question 5): Coverage of urban/driving scenes is limited

We do not explicitly cover urban/driving scenes and their associated objects, though we do expect our results to generalize to new situations such as these. We evaluate the robustness of the geometry estimation of particular objects, with our selection of objects encompassing different geometric attributes – thin structure (chair, desk), planar shapes (desk, cabinet), irregular curved surfaces (fish, cactus). We expect that a model that can robustly predict these geometries would succeed in an urban driving environment, though we leave this to future work.

Concern 4: “Model Scope excludes recent foundation models”

We respectfully disagree; our models are selected to encompass recent state-of-the-art advances in monocular depth estimation and geometry estimation. Of the 9 models we evaluate, 3 were presented at conferences or published on arxiv.org in 2025, and an additional 4 were presented or published in 2024. This includes the diffusion model Marigold and models which predict multiple modalities – Depth Pro, UniDepthV2, MoGe, and Metric3DV2.

We additionally present results for Geowizard [Fu et al, 2024] and VGGT [Wang et al, 2025] (published March 14, 2025, concurrent with NeurIPS submission). Geowizard extends the Stable Diffusion [Rombach et al, 2022] model to predict depth + normal, while VGGT predicts a point cloud from multiple input images.

We find that neither Geowizard nor VGGT display results competitive with the best monocular depth estimation models. We summarize the results below and will include full results in the final paper.

Concern 5: “There is no analysis of computational efficiency vs. robustness tradeoffs”

In figure 8 (L277) and section 4.3 (L279-286) we analyze the robustness of various models compared to their FLOPs.

We additionally measure the runtime for four of the top-performing models on an RTX 3090 GPU: UniDepthV2 (94 ms), MoGe (96 ms), DepthAnythingV2 (200 ms); and DepthPro (900 ms). We will include complete data in the final version of the paper.

Question 1 and 2: Can you describe the rationale for selection of perturbation types and parameters (e.g. occlusion percentage)?

We choose to focus on perturbations that alter the 3D composition of the scene (e.g. object/camera pose, object structure, materials). These typically require re-rendering the original scene and thus are harder to account for via image augmentations at training time. This is in contrast to motion blur or weather effects which are typically implemented as 2D transformations to the image’s pixels.

We choose the parameters of the perturbations so that the depth remains perceptible to a human and the changes to the ground truth are small. For occlusion we observed that 10-40% allowed the object’s shape to remain perceptible but it became difficult at higher percentages. For Camera Dolly Zoom and Object Resizing, we move the camera in order to maintain the 2D size of the object. This preserves the visual information and complexity of the task.

Question 3: Did you consider alternative metrics (e.g. scale-invariant error) to include affine-invariant models in consistency analysis?

Unfortunately scale-invariant metrics are not sufficient, it must be affine-invariant. Another alternative is to fix a shift and scale, but this can substantially impact the geometry and there is no clear standardization. We additionally plan to evaluate self-consistency of affine-invariant models using the affine-invariant relative depth metric between pairs of pixels, which has been adopted by [Yang et al., 2024, Depth Anything V2] and [Chen et al., 2016, Single-Image Depth Perception in the Wild]. Specifically, for model’s prediction of perturbed data d1, and base data, d2, we randomly sample 10^6 pairs of pixels from the object. We report the ratio of pairs such that 1[d1(i)<d1(j)] = 1[d2(i)<d2(j)] (where 1[] is the Iverson bracket notation, (i, j) is one pair of sampled pixels). We will include detailed results in the final version.

Question 4: AbsRel error may miss local inconsistencies. Did you consider edge aware metrics to evaluate geometric fidelity?

We considered using edge aware metrics (e.g. edge f1 score) but this conflicts with our notion of evaluating the metrics on the object of interest since the edges necessarily include portions of the background. In the appendix (table 3) we include RMSE which also captures local geometric accuracy on the object of interest.

Question 6: Application to robotics/AR systems

In robotics/AR systems, models can use richer information to improve depth estimation. For example Calibrated cameras with known focal lengths can mitigate ambiguity in dolly zoom. Using depth sensor readings, stereo input, or videos has also been shown to improve depth estimation [Chen et al., 2024, Video Depth Anything]

Question 7: Support for custom scenes

We support Blender compatible scenes, but non-rigid object deformation requires procedural assets.

Question 8: Could this framework evaluate video-based depth estimation?

Yes. Our perturbations can also be applied to generate perturbed videos. Video-based depth estimation metrics (e.g. temporal inconsistency) can be computed and used to calculate the robustness.

Thanks for your constructive feedback, and we are glad that you found our benchmark novel with “Comprehensive and Controlled Evaluation” and “Thorough Evaluation of SOTA Models”. We have addressed the individual concerns below.

Concern 1 (and question 1): Results on synthetic data may not generalize to the real world

While we acknowledge concerns about the domain gap, we argue that synthetic evaluation provides invaluable insights that complement measurements with real-world data. Real-world depth faces fundamental limitations that are addressed by synthetic data: (1) challenges obtaining dense ground truth depth in the presence of specular surfaces, transparent objects, and distant objects (2) limited scene variety (e.g. no underwater scenes) (3) prohibitive costs when creating controlled and precise perturbations at a large scale. In the context of our experiments, synthetic data can effectively reveal flaws in models’ performance by enabling precise comparisons for many scenes and perturbations. Notably, this includes perturbations such as object deformation which cannot be easily replicated in the real world.

In addition, our synthetic images are legible to humans in terms of their 3D structure. The fact that models struggle on them reveals potential misalignment of models with human perception, which can help guide model development.

To address this question about generalizability we performed preliminary real-world experiments using a mug with a known 3D structure. We then captured 15-20 images per variation and manually annotated the pose and object mask to infer a precise depth map for the object. We present the results below. Since this is a limited experiment in the context of objects on a table top (similar to the domain of robotic arms), one should not expect the results to perfectly match our much more diverse synthetic evaluation.

Error

Stability

We find similar though not identical trends in terms of models’ average performance, with UniDepth, DepthAnythingV2, and DepthPro remaining as 3 of the top 4 models by average error. We find that occlusion remains a difficult task and many models remain fairly robust to changes in lighting. We will include a more thorough analysis in our final version.

Concern 2: There is no coverage of urban driving scenarios where models may exhibit different behaviors.

We do not explicitly cover urban/driving scenes, though we do expect our results to generalize to new situations such as these. We evaluate the robustness of the geometry estimation of particular objects, with our selection of objects encompassing different geometric attributes – thin structure (chair, desk), planar shapes (desk, cabinet), irregular curved surfaces (fish, cactus). We expect that a model that can robustly predict these geometries would succeed in an urban driving environment, though we leave this to future work.

Concern 3 (and Question 2): Perturbations are tested in isolation, so the interactions between perturbations are unknown.

We believe testing individual perturbations is sufficient for evaluating robustness. Real-world scenes are not simply “clean” scenes with perturbations added; they naturally contain a mix of challenging factors. Since our base scenes already exhibit substantial diversity, applying perturbations individually already captures a wide range of realistic conditions.

Some of the perturbations (in-distribution perturbations) already exist in the base scenes, e.g., diverse lighting conditions, materials, and camera tilt angles; some of them (out-of-distribution perturbations) do not, e.g., camera roll angles are typically standardized in the base scene. For those in-distribution perturbations, we expect that combining perturbations will have at most a small impact on the average error, while the instability may increase due to the greater difference between the perturbed scene and the original. When combining out-of-distribution perturbations, we would expect the average error to increase, though the exact effect is unclear.

We present preliminary results for combinations of perturbations. Specifically, we generate the variations Camera Pan/Tilt + Translate Object, Camera Roll + Object Material Swap, and Translate Object + Scene Material Swap for chairs and desks. The following tables show the results averaged over all models for the original perturbations and the combined perturbations for the objects chairs and desks.

This confirms our expectations about the interactions between variations. For variations that introduce out-of-distribution changes, such as camera roll + material swap, the average error compounds when perturbing the scene with both variations. For object translation, which produces in-distribution perturbed scenes, the average error is close to the maximum of the two original errors. We do find for Cam Pan/Tilt + Translate Object that the instability increases, which is sensible since the difference from the original image is now larger.

Concern 4: The paper does not report error bars or statistical significance for comparisons. Incorporating error bars in plots or statistical tests would better support claims that certain perturbations are ``much more difficult'' than others, beyond just avg values.

In this case, it is unclear exactly how we could report meaningful error bars. We could attempt to understand the randomness in the models’ scores with respect to the data generation process by generating different versions of our dataset. However, we believe this to be less useful than spending time generating and verifying the quality of a larger dataset. Alternatively, could compute error bars for the average perturbation difficulty (error or stability) while treating the model outputs as samples. However, these error bars would neglect the relations between the model predictions for different perturbations. Since we currently have only 9 models we don’t believe these error bars would provide much insight.

Question 3: Why are camera perturbations particularly challenging and why are some models more robust to camera changes?

It is somewhat surprising that camera perturbations are challenging, and the success of UniDepthV2 and DepthPro suggests that an explicitly camera model is important to achieve robustness. For camera dolly zoom, we see a large difference in stability between the top models (Depth Pro and UniDepthV2) compared to the average of all models, while such phenomenon is not seen for other perturbation types. Depth Pro and UniDepthV2 both explicitly handle camera models. DepthPro predicts canonical inverse depth, and UniDepthV2 disentangles camera and depth prediction. Notably UniDepthV2's predictions of camera roll are much more stable than DepthPro. This might suggest that predicting dense camera representation is better than sparse representation (focal length).

Question 4: What are some insights or hypotheses or improving model robustness to the difficult perturbations? Would data augmentation or certain model architectures help?

We suspect that explicit augmentation would improve performance for Camera Roll, as such data is fairly rare. For other challenging variations (e.g. material changes or occlusion) we expect it would be more valuable to utilize challenging high quality data sources which may be real or synthetically generated. Another possible hypothesis is that a local geometry loss similar to MoGe would improve model’s robustness, as MoGe performs exceptionally well on the self-consistency score. However, its average error still falls behind that of Depth Pro.

Formatting: Figure 1 fails to load in Safari

Thanks for pointing this out. We have adjusted figure 1 and verified that it works on all major desktop and mobile browsers.