3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation

We appreciate that Reviewer jg4A acknowledges our promising results and the novelty of our method. We hope the following response will address the reviewer's concerns.

Q.1 Object Detection. In this paper, we follow the setting commonly used in previous methods [1][2][3] that focus on object rotation estimation. Nevertheless, we do consider the practical application scenario where the object bounding box is noisy. Specifically, we evaluated the robustness to this noise, and the results in Fig. 4(b) demonstrate the promising robustness of our method. Note that some of-the-shelf techniques can be used to achieve unseen object detection, such as SAM or the translation estimator presented in Gen6D. In Fig. 4 of the supplementary material, we show examples of 6D relative object pose estimates obtained by combining our method with Gen6D's translation estimator.

[1] Learning descriptors for object recognition and 3d pose estimation. Paul et al., CVPR 2015.

[2] Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. Xiao et al., BMVC 2019.

[3] Fusing local similarities for retrieval-based 3D orientation estimation of unseen objects. Zhao et al., ECCV 2022.

Q.2 Ablation studies. Thanks for pointing this out. RelPose* is actually the baseline suggested by Reviewer jg4A. It aims to assess the major contribution of our paper, i.e., the 3D-aware hypothesis and verification mechanism. To this end, we maintain the backbone and the 3D representation learning module in our method unchanged, solely replacing the pose-aware verification module with the energy-based module used in RelPose and RelPose++. We denote this as RelPose*. As shown in Table 4 of the main paper, the performance of RelPose* is much worse than that of our approach, confirming that pose-aware verification is pivotal in our framework.

Q.3 Typos. Thanks! We have corrected them.

Q.4 Number of hypotheses. We use much fewer pose samples than RelPose and RelPose++, i.e., 50,000 for us vs. 500,000 for them. The decision to use fewer samples during training than testing was driven by the computational resource requirements for enabling mini-batch training. We evaluate the computation cost as multiply-accumulate operations (MACs) and show the results on CO3D in Table 1 of the supplementary material. Our method achieves a better trade-off between efficiency and accuracy in relative object pose estimation, delivering more accurate results with only 5,000 samples than RelPose++ with 500,000 samples.

We thank Reviewer jWeY for the suggestions and questions. Below are our responses to each question.

Q.1 Single-reference setting. As discussed in Sec. 2 of the main paper, existing methods for generalizable object pose estimation depend on densely sampled reference images, limiting their applicability. For instance, in scenarios such as pose estimation for non-cooperative satellites, obtaining images with diverse object poses as references is often impractical. Furthermore, in many applications, e.g., autonomous driving, only sparse viewpoints are available as references, resulting in significant object pose differences between the query and its nearest reference. In such cases, our method remains applicable since it yields promising results with a single reference, where large object pose variations are involved. Importantly, by considering the single-reference case, our work can be thought of as tackling a more general problem in the field of generalizable object pose estimation. We therefore hope that our insights will inspire future research on this important topic.

Notably, predicting the relative object pose remains challenging when there is no overlap between the visible regions in the query and the reference. This issue has been thoroughly discussed in Sec. 2 and illustrated in Fig. 2 of the supplementary material. To enhance the fairness of the presented evaluation benchmark, we excluded image pairs with inappropriate references from both training and testing sets. Please refer to the supplementary material for the details.

Q.2 3D volume representation. Unlike the mesh reconstruction task where the goal is to explicitly reconstruct a 3D object mesh from images, the 3D volume representation learned in our method serves the purpose of enabling 3D-aware pose verification. The spatial dimensionality of this representation aligns with that of the sampled rotation matrix, facilitating 3D transformations over the learned volume. Consequently, the subsequent verification is conditioned on a pose sample. In this context, pose-aware features are learned during training, making it challenging to visualize such deep features. To shed more light on the pose-aware features, we provide a detailed illustration of the verification scores for the sampled pose hypotheses in Fig. 5 of the supplementary material. In this figure, the x-axis represents the geodesic distance between the pose sample and the ground-truth relative object pose, while the y-axis indicates the verification scores. The plot reveals that the predicted score decreases as the geodesic distance becomes larger. This observation shows that our method is capable of capturing pose-aware features from the images.

Q.3 Failure cases. As discussed in Sec. 5 of the main paper, our method may be affected by occlusions. Specifically, when computing the verification score, we estimate the local similarities of 2D feature embeddings, which preserve local geometric information. In instances of occlusion, however, some of the local information becomes unreliable. We acknowledge this as a limitation of our method and we plan to address this issue in our future work.

Q.4 Backbone. We use MiDAS as the backbone for our method. To ensure a fair comparison with previous methods such as RelPose and RelPose++, we conducted an ablation study in Sec. 4.5 of the main paper. We replaced the introduced pose-aware verification module with the energy-based module used in RelPose and RelPose++, while keeping the backbone and the 3D representation learning module in our method unchanged. We denote this variant as RelPose* and report the results in Table 4 of the main paper. The performance considerably drops after the replacement, highlighting the critical role of the pose-aware verification module in our method rather than the backbone.

Q.5 Efficiency. During testing, our method utilizes 50,000 pose samples, while RelPose++ uses 500,000. Despite processing fewer samples, our method achieves better accuracy in relative object pose estimation. Both methods handle the samples in parallel. The computation cost, measured in multiply-accumulate operations (MACs), is reported in Table 1 of the supplementary material. Additionally, we assess the speed of the methods on an A100 GPU. On average, our method and RelPose++ process a pair of images in 35ms and 29ms, respectively. It is worth noting that some inefficiencies, such as the use of 'for' loops, exist in our current implementation, which may impact the speed.

I appreciate the authors for providing detailed responses to all my questions. I have no further questions at the moment and would like to keep my previous positive rating score.

We are glad to see that Reviewer c3Qb recognizes the soundness of our approach and the promising results achieved by our method.

Q1. Generalization. We appreciate the insightful suggestion. Since the presented benchmark includes two datasets, we did a cross-dataset experiment at the early stage of this project. This involved training a network on synthetic data from Objaverse and subsequently testing it on LINEMOD. The acc @ $30^{\circ}$ of our method and that of RelPose++ are 21.2% and 16.7%, respectively. Our method still achieves better performance in relative object pose estimation. However, the accuracy of both methods is lower than the results reported in our main paper. We attribute this discrepancy to factors such as the synthetic-real gap, and differences in camera intrinsic parameters and resolutions across datasets, rather than to the unseen objects. This observation aligns with discussions found in some related studies. For example, Gen6D suggests training the network on multiple datasets. In their approach, some objects from LINEMOD are chosen as training data, and all images depicting these objects are excluded during testing to ensure the scenario of unseen object pose estimation. To shed more light on this finding, we mimicked the setting in Gen6D by fine-tuning our method for only 200 steps on the same subset of LINEMOD using the model pretrained on Objaverse. This took only around two minutes. Note that, in this case, the objects from LINEMOD during testing are still unseen. The acc @ $30^{\circ}$ significantly increases from 21.2% to 60.1%. While these results suggest that fine-tuning effectively mitigates the domain gap, we agree with the reviewer that cross-domain generalization is an important and interesting research problem. We nonetheless leave it as future work.

Q.2 Running time. Thanks for the suggestion! Compared to RelPose++ which requires 500,000 pose samples, our method only uses 50,000 while achieving much better accuracy in relative object pose estimation. In response to the reviewer's suggestion, we assessed the runtime performance of our method and RelPose++ on an A100 GPU. Both of these two methods process the pose samples in a parallel manner. On average, our method and RelPose++ take 35ms and 29ms, respectively, to process a pair of images. Due to some inefficiencies in our current implementation such as 'for' loops, our method is slightly slower than RelPose++. However, the sacrifice in running time is deemed acceptable given the superior accuracy achieved by our method.

We sincerely thank Reviewer nmWB for acknowledging our technical contributions. Below are detailed responses to the reviewer's concerns:

Q1. Single-reference setting. It is indeed often impractical to assume the availability of the textured 3D mesh model for a novel object, and we therefore acknowledge the contributions of the model-free approaches, including the ones listed by the reviewer. The missing references have been added to the main paper. To reasonably define the pose of an unseen object, these methods rely on multi-view calibrated reference images. Let us take Gen6D as an example since methods such as Pose from Shape [B] and OSOP [E] still require the 3D object model to generate the references. Gen6D exploits real reference images that are calibrated using a structure-from-motion technique. The goal then is to estimate the relative object pose between the query and the most similar reference selected from all multi-view reference images. Our method addresses a more general and challenging scenario where only a single reference is available. In this case, the potentially large object pose difference between the query and the reference makes methods such as Gen6D inapplicable, whereas our method can still predict the relative object pose. Moreover, in our experiments, we show that, when the reference object pose is known, such as in the CO3D, Objaverse, and LINEMOD datasets, we can exploit this information to derive the query object pose. Nevertheless, our method can still be applied to scenarios where the reference is not calibrated. For example, RelPose and RelPose++ introduce a scenario of sparse-view 3D reconstruction for unseen objects, where our method can serve as a stepping stone. In this context, one view act as reference and its rotation is set to the identity matrix. The pairwise relative transformations obtained by our approach can be leveraged to reconstruct the object's 3D structure.

Q2. Translation estimation. Rotation estimation is acknowledged to be more challenging than translation estimation because it involves computing an element $\mathbf{R}\in SO(3)$, and as such, it has received more attention [1][2][3]. As discussed in Gen6D, object translation estimation is facilitated by 2D object detection, and zero-shot object detection can be achieved using Gen6D's detector or other techniques such as SAM. In our paper, we conducted an experiment to evaluate the robustness of our approach to the object bounding box. As shown in Fig. 4 (b), our method achieves better robustness than other evaluated approaches, thus showing its potential to be integrated with such object detectors. In Fig. 4 of the supplementary material, we illustrated some 6D object pose estimation results, combining our method and the translation estimator introduced in Gen6D.

[1] Learning descriptors for object recognition and 3d pose estimation. Paul et al., CVPR 2015.

[2] Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. Xiao et al., BMVC 2019.

[3] Fusing local similarities for retrieval-based 3D orientation estimation of unseen objects. Zhao et al., ECCV 2022.

Q.3 Efficiency. Our method requires much fewer pose hypotheses than the state-of-the-art RelPose++ while achieving more accurate results. Specifically, RelPose++ samples 500,000 poses during testing while we only use 50,000. For both RelPose++ and our method, we compute the verification scores for all pose samples in a parallel manner during testing, so we measure the computation cost as multiply-accumulate operations (MACs) and report the results on CO3D in Table 1 of the supplementary material. The results clearly show that our method achieves a better trade-off between efficiency and accuracy in relative object pose estimation. Our method with only 5,000 samples still delivers more accurate results than RelPose++ with 500,000 samples. The average processing time on an A100 GPU for our method and RelPose++ is 35ms and 29ms, respectively. It is important to note that our current implementation contains certain inefficiencies, including the use of 'for' loops, which may influence the overall speed.

Q.4 Comparison on CO3D. All evaluated methods were compared fairly. We strictly followed the setting in RelPose++, training and testing our method on the same data as RelPose and RelPose++. The results reported in Table 1 of our main paper are comparable to the ones shown in Table 1 of RelPose++. On Objaverse and LINEMOD, we also conducted a fair comparison, using the same training and testing data for all methods.

Q.5 Minor details. Thanks for pointing this out. We have updated our paper accordingly.