RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions

Review (W1): My primary concern is the practical applicability of this problem. Is there a significant use case for depth prediction using text descriptions as input? For autonomous driving or AR applications, the user might not be always speaking and describing the environment.

Response (W1): Our method predicts the scale for an entire 3D scene based on a text description, as the scale is closely related to the objects and their typical sizes within the scene. In practical use, a human only needs to provide a description once upon entering a new 3D scene. As long as the surroundings do not change dramatically within that scene, there is no need for repeated descriptions.

Review (W2): For the case of automatically generating text from a detection model, essentially, this is leveraging prior knowledges from the pre-trained detector and CLIP model. What if you just extract the detector/CLIP features as input to the RSA model?

Response (W2): The comparisons are presented in the paper. In Tables 1, 2, 3 in the main paper, the "Image" entry under the "Scaling" column represents the use of CLIP image features as input for the RSA model. The findings show that predicting scale using images is less robust, especially when trained on both indoor and outdoor datasets simultaneously and when evaluated by zero-shot transfer to other unseen datasets. This is due to the larger domain gap for images compared to text description across different scenes.

Review (Q1): It's better to have more comparison with other works that are trying to retrieve metric depth scaling factor.

Response (Q1): To the best of our knowledge, we have compared all works or settings that retrieve metric depth scaling factors. If the reviewer has specific works in mind, we are happy to make comparisons. Works producing relative depth like Depth Anything and Midas conduct evaluation by doing linear fit to fit the optimal scale and shift for relative depth and ground truth, or using the median value of depth ground truth to calculate scale to conduct median scaling. We treat both of these settings as oracles of our model, since they use depth ground truth to calculate scale and shift. Also, we compare with a domain adaptation method ZeroDepth[16] in Table 1, 2, 3 in the main paper, which retrieves metric depth scaling factor to adapt a pre-trained depth model to a new domain. Additionally, in Tables 3, 4 attached, we provide a baseline of scaling the reconstruction based on 3D object sizes and their projections onto the image by directly solving for scale specific to each image. We achieve better performance across various base relative depth models, metrics, and datasets compared with ZeroDepth and the scaling baseline.

Review (Q2): As mentioned above, in my understanding, this approach is leveraging prior knowledges from the pre-trained detector and CLIP model. The text generation and encoding is more like a human-designed bootstrapping of "how to extract prior knowledge". It would be interesting and greatly helpful if the author could have comparison to direct fusion of detector/CLIP feature into the RSA model to prove the effectiveness of this bootstrapping.}

Response (Q2): To analyze the effect of prior knowledge from the detector (Detic[48]) we use, we present the experiment results labeled "Detector feature" in Tables 5, 6 attached. To fuse all proposal features, for each image processed by the detector, we pass each proposal feature through a 3-layer MLP, then perform average pooling on all proposal features to obtain a global detection feature of the image. This detection feature is then used as input to train our RSA model to predict scale. The results indicate that prior knowledge from the detector does not improve scale prediction and performs worse than using text descriptions as input. When jointly training and evaluating on both indoor and outdoor datasets, the domain gap between indoor and outdoor detection features is larger than that of text descriptions, weakening its generalization ability across different scenes.

Review (L1): The text description of images could be hard to obtain in autonomous driving or AR scenarios, limits the potential application of this method.

Response (L1): Since our method estimates the scale of an entire 3D scene from a text description. In practical scenarios, a human only needs to describe the scene once upon entering a new 3D environment. This approach is user-friendly and does not require extensive text descriptions. Provided that the surroundings remain relatively unchanged, there is no need for repeated descriptions.

Review [1]: Robustness of the method. Despite various design choices and sample images are discussed in the paper, it is not clear how well the method generalizes to difficult images, where there is less context or objects present (e.g. an empty corner of a room, or empty section of a street), or when the off-the-shelf object detectors make mistakes. In this case, methods which rely on intermediate representations including RSA may not be as robust as methods that directly operate on the image appearance.

Response [1]: Indeed, we consider this in Table 5 of the main paper, where we deleted, replaced, and introduced erroneous descriptions. By design of augmentations, our method is robust to typical noise and errors in the text, such as mis-detections, deletions, and swappings. Rows marked with "Background only" also show that our model is able to operate only background and without any objects detected. However, as discussed in the limitations (L277-287), by offering flexibility in use through captions, our method also is susceptible to malicious users who may explicitly choose completely incorrect descriptions to alter the scale of the 3D scene. This is also shown in row 8 of Table 5, where one may completely disregard indoor objects present in the image, e.g., coffee table, chair, etc., and provide outdoor objects, e.g., car, traffic light, tree, etc. Our model naturally scales the reconstruction larger, offering user controllability of the scale of the 3D scene relevant to 3D artists, but does not score well on the evaluation protocols under this scenario. In light of the review, we experimented with including background (obtained from panoptic segmentation using MaskDINO) as part of the descriptions and found that it improves results even more (see rows marked with "Object + background" in Table 5, 6 attached).

Review [2]: Limited evaluation. Despite evaluation is done on various dataset on different settings, it still feel the scale of evaluation is limited, given the claim that the proposed method is not domain specific. In this case, there should ideally be more datasets involved to represent domains in the wild (e.g. indoor/outdoor/driving, real/synthetic), yet the paper only picks a few to evaluate on.

Response [2]: Thanks for the advice. Apart from those three datasets (NYUv2, KITTI, and SUN3D) we evaluated in the main paper, we now have provided additional zero-shot generalization experiments on DDAD (outdoor) and VOID (indoor) in Tables 1, 2 attached. Our conclusions from the main paper remain valid.

Review [3]: Simple model design and prompt design. The paper picks a very simple MLP-based model without discussing on alternative choices. Moreover, the prompt design can be further elaborated, to include e.g. coordinates of objects, intrinsics of the camera, configuration of the objects in the scenes, as those properties may add additional information on the scale of the scene and to help further disambiguate the issue (for instance, existing methods for geometric estimation are known to perform worse when the FoV of the camera changes).

Response [3.1]: The RSA model needs to map text features (typically of dimension 1024) to two scalars (scale and shift). This mapping is relatively simple, and using an overly complex model like Transformers might lead to overfitting. Our comparison of different model designs, as shown in Tables 5, 6 attached, indicates that a simple 3-layer MLP is more robust and generalizable when jointly trained on both indoor and outdoor datasets, effectively avoiding overfitting. For fair comparison, We use Transformer or MLP to map input features (dim=1024) to hidden representations (dim=256), then use two separate 3-layer MLPs to map hidden representations to two scalars (scale and shift). For Transformer, we set the number of attention heads to 4.

Response [3.2]: Regarding prompt design, we conducted experiments with various text prompts, as shown in Tables 5, 6 attached. We incorporated object coordinates, camera intrinsics, and scene configurations into the text descriptions. For object coordinates, we used the center point, e.g., "a bed located at (124,256) occupying 13.42% of the image." For object configurations, we divided the image into 9 grids (top left, top, top right; left, center, right; bottom left, bottom, bottom right) and selected the grid where most of the object falls, e.g., "a bed located in the bottom left occupying 13.42% of the image." Our findings indicate that overly complex prompts can introduce noise and misguide the model. Additionally, in real-world scenarios, humans typically do not provide highly complex text descriptions. Therefore, we limited the complexity of text descriptions to include only the observed size and type of objects.

Review [4.1]: What is the method termed RSA?

Response [4.1]: It stands for "Resolving Scale Ambiguities" in the paper title.

Review [4.2]: L199, how does the model take CLIP image and text features respectively? What are the dimension of features and layers of the model in both cases?

Response [4.2]: We use CLIP ResNet-50 model, and extract image features and text features using its visual encoder and text encoder respectively. We use the final feature after global pooling, whose dimension is 1024.

Review: The proposed solution seems unreasonable... LLM does not utilize the image but only text alone.

Response: There may be a confusion. We do not use an LLM, nor do we mention LLMs in the main paper. The proposed study investigates the hypothesis of whether language, in the form of captions or descriptions, can infer the scale of 3D scenes. This bears both scientific and practical merit. Scientifically (see L20-38), this addresses the long-standing problem of scale ambiguity in monocular depth or structure-from-motion problems, where its ill-posedness necessitates a prior, or an inductive bias. We posit that objects observed in the 3D scene provide sufficient cues to infer the scale (e.g., order of millimeters, meters, 10s of meters, etc.) of the reconstruction. To the best of our knowledge, we are the first to use language to infer scale for monocular depth estimation (L76-77). Practically (see L46-50), text is arguably cheaper to obtain than range measurements (e.g., from lidar, radar) and is invariant to nuisances (e.g., illumination, occlusion, viewpoint) present in images. This allows one to generalize across domains, as evidenced by our zero-shot evaluation in Tables 1, 2 attached and Table 3 in the main paper. Our work also opens avenues to forgo additional sensors.

Review: This concern is also shown in the evaluation, were only a limited evaluation on 2 datasets is performed. Results are insignificantly improved.

Response: We clarify that we tested our method on three datasets: NYUv2, KITTI, and SUN3D. Nonetheless, per the reviewer’s suggestion, we evaluate our method on two additional datasets: VOID (indoor) and DDAD (outdoor) (see Tables 1, 2 attached). Our conclusions from the main paper still hold.

Review: The comparison is limited so simple baselines, while work that explicitly uses object scales to refine relative depth are not shown. (Only few datasets, Insignificant improvements)

Response: As stated in L76-77, to the best of our knowledge, we are the first to consider using descriptions of objects to infer scale. The closest work is ZeroDepth [16] (excerpt from [16], “geometric embeddings ... to learn a scale prior over objects”), which we compare with in Tables 1, 2 in the main paper. If the reviewer has specific works in mind, we are happy to make comparisons. Nonetheless, in Tables 3, 4 attached, we provide a baseline of scaling the reconstruction based on 3D object sizes and their projections onto the image by directly solving for scale specific to each image. Our method improves over explicitly using known object sizes to scale the reconstruction for both KITTI and NYUv2.

Review: …limited to existing object detection... The value of extracting text from images to then compute scales from text seems limited in real applications.

Response: As stated in L48-50 of the main paper, an object detector is only used for ease of experiments; there is nothing in our method that ties us down to it. Tables 5, 6 in the attached show that segmentation models can also be used to construct captions (see rows marked with ``Objects + Background''). In practice, assuming spatial continuity, one only needs to describe the 3D scene once and use the inferred transformation parameters for the rest of the scene. This facilitates metric-scale depth estimates in sensor heavy applications such as autonomous driving and augmented, mixed, and virtual reality.

Review: A comparison to finetuned GPT4v (or equivalent) with image data alone to directly estimate scales is suggested.

Response: We do provide this in Tables 1, 2, 3 of the main text. Rows marked with "Image" in the "Scaling" column refer to using CLIP image features alone to directly estimate scale.

Review: Code not yet available

Response: We will release our code upon acceptance.

Review: The paper could benefit from a deeper theoretical analysis of why the proposed method works

Response: Our method predicts scale from language description to ground relative depth estimation to metric scale. Theoretically speaking, our hypothesis is that certain objects (e.g., cars, trees, street signs) with certain observed sizes are frequently associated with specific scene types and sizes (see L40-44), leading to clusters of these object features in the latent space. The centers of these clusters can then be mapped to a 3D scene by attributing them to depth. By assuming that scene features can be mapped to a scale and shift, our models aim to learn the mapping from object semantics to the corresponding scale and shift through the context of scenes.

Review: …and its limitations;

Response: We discussed limitations in L277 - 287 in the main paper. At present, the text inputs we evaluate are generated from detection results, but this is to facilitate experiments. Future research could consider more varied language inputs, including text contains background description (see Tables 5, 6 attached) and those generated by humans. Also, factors such as occlusions, rotations, and varying object sizes, which are only partially captured in current language descriptions, can impact RSA's accuracy in predicting scale information. Moreover, while a simple global scale has been effective, it may not always be sufficiently precise for converting relative depth to absolute depth, especially when the predicted relative depth is inaccurate. In such situations, global transformation parameters might fail to adequately scale the depth map as the assumption that reconstruction is up to an unknown scale is violated. Future studies could expand RSA to accommodate finer adjustments and explore the possibility of inferring region-specific or pixel-level scales using language inputs. Finally, although language inputs are highly user-friendly, RSA remains susceptible to manipulation by malicious users who might provide misleading captions to distort predictions.

Review: Additional experiments on a wider range of datasets and scenes would strengthen the evidence for the method’s generalizability and robustness;

Response: Thank you for the suggestion. We have provided additional experiments on DDAD and VOID in Tables 1, 2 attached. Additionally, in Tables 3, 4, 5, 6 attached, we provide experiments on using 3D object sizes, background together with objects, background only, detector features, coordinates and configuration of objects, camera intrinsics, as well as different architecture design choices. Our conclusions from the main paper still hold.

Review: The description of the linear transformation process and the role of text captions need more detail to fully understand the approach.

Response: For each input image, RSA predicts two scalars: scale and shift, given the text description. Then, as shown in Figure 2 in the paper, for a relative depth predicted by a frozen depth model, we take its inverse, multiply it with the predicted scale, and add the predicted shift, to obtain the transformed metric depth prediction (see Sec. 3). The role of text captions serves as input for the model to predict scale. The problem we would like to investigate is whether language can be used as input to infer scale. In this way, we could ground relative depth prediction from SOTA Monocular Depth Estimation models, like Midas or Depth Anything, to metric scale. In practical usage, it would be a human-computer interaction scenario, where humans provide short descriptions of the scene to ground model predictions. For experiments, we use derived text from images to simulate description that humans would provide in real-world scenarios.

Review: How does the method perform in real-world scenarios where text captions may not be readily available or may be noisy?

Response: We do test for robustness to noisy descriptions in Table 5 of the main paper. We note that RSA is trained with augmentations including random swapping and deletion of objects, so our model is robust to noisy descriptions (missing or change in order). We note that our paper is to investigate the use of text captions to infer scale, so forgoing them completely would leave us with no input. Instead, in Tables 5 and 6 attached (see rows marked with "Background only"), we present results for the more likely case that no objects are detected and the model is presented only with the background.