LabelAny3D: Label Any Object 3D in the Wild

We thank the reviewer for the constructive comments and for recognizing our “clear writing” and “novel and effective pipeline”.

** $W1$ The paper should test more models to show the effectiveness of their new dataset.**

Thank you for the constructive suggestion! Open-vocabulary monocular 3D object detection is still an emerging task. To our knowledge, only three works focus specifically on this problem: OVMono3D (which we use as our base model), OVM3D-Det (our baseline with a pseudo-annotation pipeline), and DetAny3D $1$ , recently accepted to ICCV 2025. DetAny3D’s model weights were released after the NeurIPS submission deadline (on June 25) and we have incorporated all its baseline comparisons. Due to time and resource constraints, we will include results from DetAny3D finetuned on our pseudo labels in the revised version. Since the author of DetAny3D didn’t provide evaluation scripts, it needs some engineering effort to complete. We will strive to provide the evaluation results during the discussion period.

** $W2$ Why reconstructs the whole 3D scene? Only reconstructing the 3D objects are enough to annotate the 3D object bbox.**

Great question! While we use TRELLIS for 3D object reconstruction, it generates objects in a normalized canonical space, without alignment to image pixels or the global scene. To obtain accurate 3D bounding box attributes—such as center, size, and pose—we need to map these objects back into their true positions within the scene. To achieve this, we first reconstruct a scene-level point map, and then perform 2D–3D alignment using MASt3R and a PnP solver. This alignment step is essential to recover each object’s 3D position and spatial context. More details are provided in Lines 148–159 of the manuscript.

** $W3$ Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?**

Thank you for the insightful suggestion! The table below presents a statistical comparison between COCO3D and existing 3D detection benchmarks on their respective test sets. We report the number of images, instances, categories, covered domains, and the distribution of instances across MS-COCO super categories. As an ongoing effort, we have completed annotation of all COCO validation images since the submission. We refer to this full set as COCO3D v2.

Compared to prior benchmarks—which often focus on specific domains such as indoor environments (e.g., SUN RGB-D), object-centric scenes (e.g., Objectron), or self-driving datasets (e.g., KITTI, nuScenes)—COCO3D offers broader coverage across indoor and outdoor everyday scenes. From the super-category perspective, COCO3D uniquely includes categories from food, animal, and sports, which are often absent in existing 3D datasets. We believe COCO3D can play a similar role in open-vocabulary monocular 3D detection as MS-COCO does in 2D detection—offering a diverse, real-world, and scalable benchmark for advancing the field.

** $Q1$ How to combine MoGe with Depth Pro at Line 141? Why using metric depth estimator there?**

Thank you for the great question! In our pseudo-annotation pipeline, we align the relative depth map from MoGe with the metric-scale depth map predicted by Depth Pro using a global affine transformation. Specifically, we first use MoGe to generate a relative depth map and extract camera intrinsics. Then, we apply Depth Pro to predict a metric-scale depth map for the same scene. We perform RANSAC-based linear regression to fit a robust scale and shift transformation that maps MoGe’s relative depths to Depth Pro’s metric scale. This allows us to retain the fine geometric details from MoGe while calibrating the scale to real-world distances. As a result, our final pseudo 3D annotations are based on scale-calibrated relative depths, which we found empirically to improve downstream 3D detection performance compared to using raw MoGe depths alone. This also enables a fairer comparison with baseline methods like OVM3D-Det, which rely exclusively on metric depth estimation.

It’s important to note, however, that we do not rely on metric depth at evaluation time on the COCO3D benchmark, due to the general difficulty of accurate metric depth prediction in unconstrained scenes. Instead, we introduce a new evaluation metric—Relative Layout Average Precision (Lines 235–245)—to assess how well the predicted 3D bounding boxes preserve the relative spatial layout compared to the ground truth.

We will clarify this explanation in the final version of the paper. Thank you again for pointing it out!

$1$ Detect Anything 3D in the Wild. Zhang et al. ICCV 2025.

Dear reviewer,

Thank you again for your thoughtful review and constructive feedback! Your input is invaluable in helping us improve this work. We remain committed to resolving any remaining issues. Please feel free to share any additional comments at your convenience.

We thank the reviewer for the constructive feedback and for finding our "topic important” and "well written”.

** $W1$ Concern about the generalization of the proposed approach and sensitivity of the hyper-parameters over a new dataset.**

We thank the reviewer for raising this important concern about generalization and hyperparameter sensitivity. To address this, we have evaluated our pipeline across multiple datasets without any hyperparameter adjustment, demonstrating its robustness.

In the original submission, we have validated our pipeline on the KITTI benchmark (Lines 280–283). LabelAny3D achieved 13.60 AP3D, outperforming the baseline OVM3D-Det (12.39 AP3D). In this rebuttal, we provide additional validation on the SUN RGB-D benchmark. LabelAny3D surpasses OVM3D-Det by +2.32 AP3D (41.39 vs. 39.07). We will include these results in the final version of the manuscript.

** $W2$ Adding LabelAny3D pseudo labels only leads to marginal performance improvement or degradation over the OVMNono3D Novel and Base datasets, when comparing against OVM3D-Det.**

Thank you for raising this point! Both our model and OVM3D-Det are trained on a mix of clean Omni3D labels and generated pseudo labels on COCO, which explains their similar performance on the OVMono3D Novel and Base categories—both derived from Omni3D.

However, our model demonstrates superior performance on COCO3D, demonstrating the strength of our high-quality pseudo labels in more diverse and open-world settings. Due to the underlying distribution differences, the benefit from our high-quality COCO pseudo annotations to Omni3D is naturally limited. Despite this, we still observe a +0.93 AP3D improvement on OVMono3D Novel categories, while OVM3D-Det shows a -0.5 AP3D drop on OVMono3D Novel categories.

OVM3D-Det shows slightly higher AR3D, which we attribute to overconfident predictions leading to more false positives. We note that average recall (AR3D) captures detection rate but not the accuracy of 3D localization, which is better reflected in AP3D.

We will clarify this point in the revised version.

** $W3$ Why does LabelAny3D show mixed results in Table 1? … run evaluation and compare the two methods over the same datasets employed in the OVM3D-Det paper.**

Thank you for the constructive suggestion. As noted in our response to W2, while LabelAny3D consistently achieves higher AP3D (precision), OVM3D-Det shows slightly higher AR3D (recall) due to overconfident predictions, which increase false positives. In response to W1, we have also evaluated both methods on the KITTI and SUNRGBD datasets used in the OVM3D-Det paper. These results will be included in the revised version of the manuscript.

** $Minor$ The paper seems incomplete without a conclusion section. Please consider adding that section if accepted.**

Thank you for pointing this out. Our Discussion section (Sec. 5, Lines 304-316) is intended to serve as the conclusion by summarizing our main contributions and acknowledged limitations. For clarity, we will rename it to Conclusion in the revised version.

Dear reviewer,

We appreciate your thoughtful review and constructive suggestions. As the discussion deadline approaches, we would greatly appreciate it if you could let us know whether our responses have fully addressed your concerns. Your feedback is invaluable, and we remain committed to resolving any remaining issues.

To further address the concerns in [W1] and [W3], we additionally validated our approach on nuScenes and Objectron benchmarks. LabelAny3D consistently surpasses OVM3D-Det by +3.16 AP3D on nuScenes (19.50 vs. 16.34) and +2.8 AP3D on Objectron (22.51 vs. 19.71).

We attribute this strong generalizability to two key factors: (1) LabelAny3D effectively integrates multiple 3D foundation models (i.e. TRELLIS, MASt3R, MoGe) which are trained from diverse high-quality 3D data and demonstrate strong generalizability in the wild. This enhances its robustness across datasets and hyperparameters; and (2) LabelAny3D adopts an analysis-by-synthesis paradigm that reconstructs holistic 3D scenes from monocular images in a class-agnostic manner. This enables robust 3D automatic labeling across object categories. In contrast, OVM3D-Det relies on object size priors from large language models, which struggles with categories exhibiting high intra-class variations.

Please don’t hesitate to let us know if further clarification is needed.

Dear reviewer MxxK,

Thank you for your constructive suggestions! We notice you've submitted your final rating - could you kindly let us know whether our rebuttal addressed your concerns?

In response to your initial comments, we conducted additional experiments on four established 3D benchmarks (including KITTI, SUN RGB-D, nuScenes and Objectron) to address your concerns regarding pipeline robustness and baseline comparisons and observed consistent outperformance. We also provided comprehensive clarifications on the points you raised.

We understand you may have submitted final justifications that are not visible to us during the discussion phase. As this phase comes to a close, we would appreciate any feedback on whether our rebuttal has adequately addressed your concerns. Your feedback is invaluable, and we remain committed to resolving any remaining issues.

We look forward to your reply at your convenience. Thank you again for your time and valuable feedback throughout this process.

We thank the reviewer for the constructive comments and for recognizing our “clear writing”, “theoretically significant”, and “strong 3D generalization capability".

** $W1$ Validation of the pipeline on datasets with 3D ground truth.**

We thank the reviewer for raising this important point. In the original submission (Lines 280–283), we validated our pipeline on the KITTI benchmark, where LabelAny3D achieved 13.60 AP3D, outperforming the baseline OVM3D-Det (12.39 AP3D). In this rebuttal, we provide additional validation on the SUN RGB-D benchmark. LabelAny3D surpasses OVM3D-Det by +2.32 AP3D (41.39 vs. 39.07). OVM3D-Det relies on object size priors from large language models to infer 3D bounding boxes. This strategy is particularly effective for objects with relatively consistent dimensions (e.g., cars), but struggles with categories exhibiting high intra-class variation (Lines 41-45). In contrast, LabelAny3D adopts an analysis-by-synthesis paradigm: it reconstructs the holistic 3D scene from monocular images and uses the synthesized representation to infer 3D annotations (Lines 52-54). We will include these results and extend our validation to more benchmarks in the final version of the manuscript.

** $Q1$ How should the proposed pipeline handle round objects that may lack a clear definition of orientation?**

We thank the reviewer for this insightful question! Orientation ambiguity in symmetric objects is a well-known challenge in 3D vision. For such objects, multiple orientations may yield indistinguishable appearances, making pose estimation inherently ambiguous.
Our pipeline does not explicitly resolve this ambiguity. Instead, it inherits the pose definition from TRELLIS, the 3D reconstruction model we used, which reconstructs objects in the canonical space defined in its training data, where the orientation of symmetric objects remains ill-defined.

Some recent works (e.g., Orient Anything $1$ ) address this issue by using vision-language models (VLMs) to identify symmetric objects and label them as having no meaningful pose. However, our primary focus is on tight 3D bounding box prediction, which remains meaningful even for symmetric objects. We therefore retain symmetric objects in both training and evaluation to preserve dataset diversity. This approach aligns with standard practice—most large-scale 3D datasets (e.g., KITTI, ARKitScenes) do not explicitly handle symmetry during training or evaluation.

An exception is Objectron $2$ , which incorporates pose equivalence in its evaluation metrics to account for such ambiguity. Specifically, predicted bounding boxes for symmetric objects are rotated along the symmetry axis, and the rotation that yields the maximum IoU with the ground truth is selected.

To assess the impact of symmetry on our benchmark, we manually annotated symmetric instances in 300 randomly sampled images from COCO3D. We found that symmetric objects comprise 13% of all instances, primarily in cylindrical shapes such as apples, bottles, and bowls. Using the pose-equivalent metric from Objectron, our 3D detector achieves 12.55 AP3D on symmetric objects, compared to 12.53 AP3D without the metric—a negligible difference of 0.02 AP3D.

In summary, our results suggest that our current results are robust to orientation ambiguity. We will include a detailed discussion of this limitation and our evaluation results in the final version of the paper.

$1$ Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models. Wang et al. ICML 2025.

$2$ Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations. Ahmadyan et al. CVPR 2021.

Dear reviewer,

Your constructive feedback has strengthened our evaluation, and we sincerely appreciate it!

To further address the concerns in [W1], we additionally validated our approach on nuScenes and Objectron benchmarks. Consistent with our prior results, LabelAny3D surpasses OVM3D-Det by +3.16 AP3D on nuScenes (19.50 vs. 16.34) and +2.8 AP3D on Objectron (22.51 vs. 19.71). By reconstructing the entire scene, LabelAny3D effectively works across different benchmarks without relying on per-category priors.

We hope our responses have adequately addressed your concerns, but please don't hesitate to point out any remaining issues. Your input is invaluable in helping us improve this work.

We appreciate the reviewer for the thoughtful feedback and praises, including "remarkable performance improvements" and "enhances detector performance".

** W1 & Q1 Systematic comparison of COCO3D benchmark with other benchmarks.**

We thank the reviewer for suggesting this important point! The table below presents a statistical comparison between COCO3D and existing 3D detection benchmarks on their respective test sets. We report the number of images, instances, categories, covered domains, and the distribution of instances across MS-COCO super categories. As an ongoing effort, we have completed annotation of all COCO validation images since the submission. We refer to this full set as COCO3D v2.

Compared to prior benchmarks—which often focus on specific domains such as indoor environments (e.g., SUN RGB-D), object-centric scenes (e.g., Objectron), or self-driving datasets (e.g., KITTI, nuScenes)—COCO3D offers broader coverage across indoor and outdoor everyday scenes. From the super-category perspective, COCO3D uniquely includes categories from food, animal, and sports, which are often absent in existing 3D datasets. We believe COCO3D can play a similar role in open-vocabulary monocular 3D detection as MS-COCO does in 2D detection—offering a diverse, real-world, and scalable benchmark for advancing the field.

** W2 & Q2 Analysis of larger model performance on COCO3D benchmark**

We thank the reviewer for suggesting experiments with larger model capacity. The results are provided in the following table. Both the baseline OVMono3D and our model were initially trained with a frozen DINOv2 backbone. To increase model capacity, we unfroze the DINOv2 backbone for both models, thereby introducing more learnable parameters.

After enlarging the model capacity, both the baseline and our model demonstrated performance improvements. This indicates that the backbone produces more task-related features after unfreezing. More importantly, the performance gap between our model and OVMono3D on base categories narrowed from 2 to 1.3 AP points. This suggests that with larger model capacity, models can better handle more diverse scenarios and categories, thus alleviating the catastrophic forgetting issue.

We also attempted to enlarge model capacity by replacing the backbone from DINOv2-base with DINOv2-large. However, the required training time exceeds the rebuttal period given our computational resources. We will strive to include this analysis in our revised version.

** W3 & Q3 Variance analysis across multiple runs on proposed COCO3D benchmark.**

We appreciate the reviewer's constructive suggestion! We conducted three training runs for each model in Table 1 (except for the baseline, for which we directly evaluated its publicly-released model checkpoint). The evaluation results on COCO3D are shown in the following table, with data presented in mean ± standard deviation format. These results validate that our benchmark provides consistent and reliable model rankings despite the inherent randomness in training procedures.

We also conducted multiple evaluation runs for each model using different subsets of COCO3D. Specifically, we sampled three subsets from the COCO3D benchmark, each containing 1500 images. Each model in Table 1 was evaluated on each subset separately, and we report the mean ± standard deviation of AP points in the following table. The results across the three subsets are stable, which further demonstrates the robustness of our benchmark.

We agree that including more images and instances would enhance the reliability and robustness of the benchmark. Since the time of submission, we have completed annotations for the entire MS-COCO validation set, and we plan to incorporate additional datasets such as Objects365 to further expand the benchmark.

We sincerely appreciate your valuable feedback and support. These revisions will be fully incorporated into our final version.