Vision Foundation Model Enables Generalizable Object Pose Estimation

Q1: Can the proposed method work well on unseen categories when the geometry/texture of reference and query objects are different?

A1: Thanks for the comment. We suppose that our presented experiments on Wild6D and CO3D datasets could address this concern. For Wild6D evaluation, we have tested our method on 162 different object instances in 5 unseen object categories. For CO3D evaluation, we have tested our method on 200 different object instances in 20 unseen object categories. In Fig.2 of the PDF file uploaded in the global response, we present some examples of reference and query objects used in Wild6D and CO3D evaluations.

• In terms of the texture difference, the reference object that we used is untextured, while the query objects in various testing scenarios usually have colorful and diverse textures. The texture difference between reference and query objects is significant.
• In terms of the geometry difference, since the precise CAD model for each object instance in Wild6D and CO3D is not available, we are not able to measure the geometry difference quantitatively. However, as depicted in Fig.2 of the uploaded PDF file, the geometry difference between reference and query objects is rather apparent. For example, the reference ‘mug’ can be used to estimate poses for a variety of mugs with different rim shapes and handle styles. Moreover, the reference ‘chair’ with four legs and no armrests can be used to estimate poses for various chairs with different leg structures and chairs with or without armrests.

We suppose that these results can demonstrate that our method can work well on unseen categories when the geometry/texture of reference and query objects are different.

Q2: Why is a two-stage approach necessary?

A2: Thanks for the comment. In our ablation study, we have validated our designs on the first and second stages. Table 4 of our original paper demonstrates the significant performance improvement brought by each individual module. If we only use the first stage, the final pose accuracy would be sensitive to the number/pose distribution of sampled reference images. To explore this problem, we evaluated our method with different numbers of reference images. These result has been presented in Sec.F of the paper appendix. As shown in Fig.14 of the appendix, the second stage makes the final pose accuracy become robust to the selection of reference images. Moreover, Fig.14 also comprehensively compared the pose accuracy after the first stage and the second stage. The second stage can consistently improve the performance. These results demonstrate the advantage and necessity of the two-stage design.

Q3: The specific required inputs and evaluation settings used for each baseline.

A3: Thanks for your valuable suggestion. To clarify, we list the involved baseline methods one by one:

• SPD, SGPA, DualPoseNet, and GPV-Pose: RGBD-based methods. We re-trained their models and evaluated them on Wild6D.

• PoseContrast: Use RGB images during object viewpoint estimation, and further leverage depth map to estimate complete object pose. We re-trained the PoseContrast model with our constructed synthetic dataset and then evaluated it on Wild6D.

• ZSP: RGBD-based baseline. Does not involve any training module and we directly evaluated it based on its official source code.

• LoFTR and LightGlue: As mentioned in L299-300 of the original paper, we need to use both RGB images and depth maps for category-level object pose estimation. We exploited their officially provided pre-trained models during experiments.

• GeDi: Point-cloud-based method. As a generalizable local deep descriptor for point cloud registration, we exploited its robust model pre-trained on 3DMatch during experiments.

We are sorry for the possible confusion caused. We will elaborate on the description for all baseline approaches in our revised version.

Q4: Did the authors experiment with GPT-4V and Text-to-3D for generating 3D models and estimating 6D poses as shown in Figure 2? Which models were used?

A4: Thanks for your interest in our GPT-4V + Text-to-3D setting. We mainly tested this setting in practical open-world scenarios from the D3-Field dataset and RH20T dataset. Figure 9 of the main paper and Figure 15 of the paper appendix present the experiment results on various unseen object categories, including ‘shoes’, ‘flower pot’, ‘watering can’, ‘forks’, and ‘ladle’. For experiments on Wild6D and CO3D, we leveraged the collected shape templates for evaluation.

Q5: How are the 64 reference views sampled?

A5: Thanks for the comments. The procedure of preparing shape templates can be divided into the following three steps: (1) Normalize the input object CAD model to have a diameter of 1; (2) 64 camera viewpoints are sampled from a sphere centered on the normalized object CAD model; (3) Rendering the corresponding RGB-D template image at each viewpoint with Blender. At each camera viewpoint, we also randomly sample the camera in-plane rotation within a specified rotation range (i.e., [$-10^\circ$, $10^\circ$]).

Q6: Which tokens are used in foundation features?

A6: Thanks for the comment. For CLIP and MVP foundation models, we used their patch tokens. For DINO-v1 and DINO-v2 models, we developed our framework based on their ‘key’ tokens. We will incorporate these implementation details during revision.

Q7: Clarify Figure 1. How is the similarity score normalized?

A7: Thanks for your careful checking. Given one query image and $N$ reference images, we would obtain $N$ raw similarity scores. We further normalize the similarity score via softmax. Note that a global scale factor of 0.5 is applied to the normalized score for the purpose of visualization, which does not affect the similarity distribution over reference views. Sorry for any confusion caused. We will further revise Figure 1 and its corresponding caption to improve its clarity.

Dear Reviewer gYMD,

Thank you for your feedback on our rebuttal. We are very glad that our response has fully addressed your earlier questions. Much appreciate your kind support for our work. We are grateful for your willingness to raise the score. Thanks a lot!

Sincerely,

Authors

Q1: The method relies on depth maps and NOCS maps, which could limit its applicability in real-world scenarios.

A1: Thanks for your comment. To address your concern, we explore the possibility of RGB-only pose prediction for our method. Due to the character limit, please refer to Q1 in the global response for detailed evaluation results.

The NOCS map of a reference image could be directly computed based on the GT pose of the reference image. It does not require any other additional information. Please refer to our response to your Q3 for details of computing the NOCS map. In this regard, the use of the NOCS map would not be a limitation of our method in practical applications.

Q2: How to perform the object shape focalization?

A2: Thanks for your comment. The focalization step takes as input the object point cloud in the camera frame and the initial object orientation estimated during the query-reference image matching stage. It aims to transform the object point cloud into a relatively canonical and normalized space to facilitate the subsequent object shape representation. Formally, the focalization step can be formulated as $\mathcal{X}'=\mathbf{R}^{-1}\times\frac{\mathcal{X}-\bar{\mathcal{X}}}{\max{\|\|\mathcal{X}-\bar{\mathcal{X}}\|\|}}$, where $\mathcal{X}$ denotes the original object point cloud in camera frame, $\bar{\mathcal{X}}$ denotes point cloud center, and $\mathbf{R}$ is the initial object orientation estimated during the query-reference image matching stage. We will elaborate on this part more clearly in the revised version.

Q3: How to obtain the NOCS map for a reference image?

A3: Thanks for pointing it out. Each pixel in the NOCS map indicates a 3D coordinate value in the normalized object frame. For each reference image, the object scale $s$ and the ground-truth object pose $[\mathbf{R}|\mathbf{t}]$ should be known. For each pixel in the camera frame, we are able to transform its 3D coordinate into the object frame based on $[\mathbf{R}|\mathbf{t}]$ and then normalize the coordinate value with $s$ to obtain the corresponding coordinate value in the NOCS map. We feel sorry for the possible confusion and will elaborate on these details in the revised version.

Q4: Recommend rephrasing the introduction to highlight the importance and novelty of the category-level setting.

A4: Thanks for the constructive suggestion. We are glad to see that the reviewer appreciated the importance of the problem that our paper targets and the novelty of our method from the category-level perceptive. We suppose that our method could be distinguished from other existing foundation-model-based works from the following aspects: (1) We show that by using our proposed adaptable framework, we can effectively enhance the 3D representation capability of the pre-trained 2D foundation model to handle intra-class shape variations of different objects for challenging category-level object pose estimation. (2) We demonstrate that the pre-trained foundation model can be effectively adapted to the task of category-level object pose estimation with cost-effective synthetic data and also keeps high robustness in handling novel object categories. We would follow your suggestion to rephrase our paper to highlight our contributions from the category-level perspective.

Q5: For query-reference image matching, comparison with patch-level template matching strategy [1][2][3] is missing.

A5: Thanks for the comment. The first patch-level template matching method [1] you suggested is Ref [71] in our original paper (cf. L322 on page 8). We have compared this template-matching strategy with our proposed feature lifting module in our ablation study. Table 4 of the original paper reports quantitative evaluation results (i.e., ‘w/o feature lifting’ vs. Ours) and Fig.11 of the appendix presents qualitative evaluation results (i.e., column (a) vs. column (c)). We suppose that these results can demonstrate the advantage of our method over the patch-level template matching strategy. We will rephrase our presentations to present this part more clearly.

Q6: LoFTR and LightGlue are trained on scene-centric images, which may be not able to generalize to object-centric scenarios without finetuning.

A6: We appreciate you bringing this up. We have followed your suggestion and fine-tuned LightGlue on object-centric images. Specifically, we leveraged the 120 object models we had collected from ShapeNet to synthesize object-centric image pairs for fine-tuning LightGlue. For each object instance, we sampled image pairs by rotating the camera within a range of 30 degrees. We generated 200 image pairs for each object instance, resulting in a total of 24K image pairs used to fine-tune LightGlue. The table below presents the evaluation results of the fine-tuned LightGlue on the CO3D dataset.

We found that for the task of category-level object pose estimation, the improvement from the object-centric fine-tuning was relatively limited. We suppose that the limitation of these feature-matching techniques is that they focus too heavily on low-level object features, and lack a sufficiently informative representation to perceive the object's shape and semantics, which makes them struggle with the challenging problem of category-level object pose estimation.

[1] Nguyen et al. Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions. CVPR 2022.

[2] Zhao et al. Fusing Local Similarities for Retrieval-based 3D Orientation Estimation of Unseen Objects. ECCV 2022.

[3] Örnek et al. FoundPose: Unseen Object Pose Estimation with Foundation Features. ECCV 2024.

Q1: Clarify Fig.1. Why is DINOv1 used in Fig.1 and not much more performant DINOv2?

A1: Thanks for pointing it out. In Fig.1, taking DINOv1 as an example, we aim to show that the pre-trained vision foundation model is not reliably used for category-level object pose estimation. This observation is indeed consistent across different representative vision foundation models, including CLIP, MVP, DINOv1, and DINOv2 (As shown in Fig.8 of the original paper). We attribute this to their limited 3D representation capability due to pre-training with text and 2D images. Based on this, we propose a foundation feature lifting module for more reliable object pose estimation, which has proven effective with different vision foundation models. To reduce the possible confusion, we will revise Fig.1 of the main paper to include results from different vision foundation models during revision.

Q2: The recent work FoundPose[1] is not discussed in the paper.

A2: Thanks for your constructive suggestion. FoundPose[1] is a recent method for object pose estimation. Similar to FoundationPose[2], it aims to tackle the problem of instance-level unseen object pose estimation. The idea of integrating DINOv2 representation and bags-of-words descriptor is very impressive. During our submission, FoundPose is still an arXiv paper, while FoundationPose has been accepted by CVPR 2024. Therefore, we mainly focus on discussing published works while preparing the manuscript. We are also glad to find that FoundPose has been accepted by ECCV 2024 (the paper decision date is 1st July 2024). Note that our method is clearly distinguished from FoundPose by addressing a more challenging category-level estimation problem for unseen object categories. We will ensure to incorporate a discussion of FoundPose in the revised manuscript.

Q3: Comparison with the recent BOP method FoundationPose[2].

A3: Thanks for your valuable suggestion. Considering that the main focus of our paper is category-level object pose estimation for unseen object categories, we compare our method with FoundationPose on the CO3D dataset. We utilized the official source code and pre-trained models from FoundationPose, adopting the model-based setup during inference. Both our method and FoundationPose use the same category-level shape templates for object pose estimation. The following table presents the evaluation results in terms of Acc.$15^\circ$ / Acc.$30^\circ$. Due to the character limit, we only report the per-category accuracy for 6 representative categories and the average accuracy across all 20 categories:

As can be observed, our method demonstrates its superiority over FoundationPose in challenging category-level object pose estimation for unseen object categories.

Q4: The procedure of preparing the shape templates. What kind of conditions are needed and what settings are used to prepare the shape templates for the inference?

A4: Thanks for the comments. The procedure of preparing shape templates can be divided into the following three steps: (1) Normalize the input object CAD model to have a diameter of 1; (2) $N$ camera viewpoints ($N$ indicates the number of template images for inference) are sampled from a sphere centered on the normalized object CAD model; (3) Rendering the corresponding RGB-D template image at each viewpoint with Blender. Our method only needs an untextured object CAD model. During rendering shape templates, we fix the position of the lighting source on the top of the object, with a random lighting color that is uniformly sampled within $[0.5, 0.5, 0.5]$~$[1.0, 1.0, 1.0]$. We will append these implementation details in our revised manuscript.

Q5: What kind of SO(3) sampling is used in generated synthetic data?

A5: Thanks for the comments. We customized the synthetic data generation pipeline[3] that has been widely used in the BOP challenge to generate our category-level synthetic data. To generate diverse synthetic data, two aspects of pose sampling are involved: (1) Object on-surface sampling. The object would be placed upright onto a plane of the synthetic scene, and the in-plane position and orientation of the object would be randomly sampled. (2) Camera pose sampling. The camera location is first sampled around the object using the "uniform_elevation" sampling used in BOP. Then, the camera rotation is further determined by a point of interest which is randomly sampled from the scene, plus a sampled camera in-plane rotation within a specified range ([$-30^\circ$, $30^\circ$] used in our synthetic data). We will append these details related to synthetic data sampling in our revised manuscript.

[1] Örnek et al. FoundPose: Unseen Object Pose Estimation with Foundation Features. ECCV 2024.

[2] Wen et al. FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. CVPR 2024.

[3] Denninger et al. BlenderProc: Reducing the Reality Gap with Photorealistic Rendering. RSS Workshop 2020.

Q1: Ablation study on RGB-only object pose estimation.

A1: Thanks for your suggestion. Due to the character limit, please refer to Q1 in the global response for detailed evaluation results. As can be observed, the RGB-only variant achieves comparable pose accuracy on average when compared with the original RGBD-based performance. These results indicate the application potential of our method in scenarios that lack depth observation. Please also refer to Fig.1 of the PDF file uploaded in the global response for detailed qualitative visualization results.

Q2: Results under different percentages of occlusion.

A2: Thanks for the good suggestion. First, we evaluated our method under different levels of occlusion on the LineMOD-Occlusion dataset. The table below reports the average recall of ADD(-s).

Second, we conducted more evaluations on the CO3D dataset. Since scenes from CO3D are generally occlusion-free, we randomly masked out different percentages of image regions to mimic the effect of object occlusion. The table below reports the Acc.$15^{\circ}$ / Acc.$30^{\circ}$ results on 5 representative categories of CO3D.

These results show that our method is relatively robust to small and moderate occlusion percentages, and would suffer a performance drop when facing severe occlusions larger than 60% occlusion rate. Moreover, as we have discussed in the main paper, in practical application scenarios, this occlusion issue could be potentially addressed by active perception and mobile manipulation to find an occlusion-free viewpoint.

Q3: Clarify results presented in Table 1.

A3: Sorry for any confusion caused. The evaluation of Wild6D contains comparisons with two groups of approaches. To improve the clarity, we have reorganized the original Table 1 into these two distinct parts to better communicate the Wild6D evaluation findings:

• Comparison with conventional category-level approaches. For the competing methods SPD, SGPA, DualPoseNet and GPV-Pose, we first trained their models on the training data containing the testing categories, and then tested their performance on Wild6D. For our method, we did not train it on data containing the testing categories, and directly tested its performance on Wild6D. The following table presents the comparative results. Conventional category-level approaches highly rely on category-specific training. They have to train and test their models on the same sets of categories. In this case, Wild6D is not unseen to them. In contrast, without training on the five testing categories, our method significantly outperforms these category-level approaches in $10^\circ2cm$ and $10^\circ5cm$ accuracy, and achieves comparable accuracy under more strict thresholds.

  	$5^\circ2cm$	$5^\circ5cm$	$10^\circ2cm$	$10^\circ5cm$
  SPD	2.6	3.5	9.7	13.9
  SGPA	20.1	27.8	29.0	39.4
  DualPoseNet	17.8	22.8	26.3	36.5
  GPV-Pose	14.1	21.5	23.8	41.1
  Ours	19.3	21.6	34.9	44.2

• Comparison with category-agnostic approaches. The following table presents the comparative results. Note that all competing methods are not trained on testing categories. In this case, categories in Wild6D are unseen to all three methods. Our method outperforms the other two competing methods in all metrics.

  	$5^\circ2cm$	$5^\circ5cm$	$10^\circ2cm$	$10^\circ5cm$
  PoseContrast	2.3	4.7	5.5	10.1
  ZSP	9.6	12.1	16.6	23.0
  Ours	19.3	21.6	34.9	44.2