Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation

We thank the reviewer for their encouraging feedback.

W1. The majority of the figures in this paper such as Fig.2, Fig.2, and Fig.10, exhibit low quality. It would be better if the authors could consider revising them to enhance the clarity and resolution.

This point has also been raised by other reviewers. We have tried our best to address the presentation issues with our initial draft by improving the quality and clarity of the figures. We are happy to take any more suggestions to improve them even further. Please refer to the updated draft for updated and improved figures.

Q1. To my understanding, in the experiments, the baseline models are evaluated in the target domain without domain adaptation. In this context, it is reasonable that those methods cannot generalize well in the target domain. I was wondering if there are some existing domain adaption approaches that can be applied to the baseline models. The evaluation would be more convincing, comparing NeMo + 3DUDA with a competitor such as NeMo + another domain adaption method.

To the best of our knowledge, our method appears to be the only current source-free image-only UDA method for object pose estimation. It is unclear what kind of methodology we could combine with NeMo for UDA since that would be nontrivial and could be an interesting new research avenue. We do have baselines (P3D [a] and DMNT [b]), which are improvements over the NeMo model and claim to be robust to OOD as well. P3D uses synthetic and real data combinations and different training methods for robust 3D pose estimation. DMNT uses deformable meshes and neural textures for robust pose estimation.

[a] Yang et. al. - Robust category-level 3d pose estimation from synthetic data

[b] Wang et. al. - Neural textured deformable meshes for robust analysis-by-synthesis

Q2. The pre-render feature maps are generated from the source mesh. As the vertex features are not updated here, how to make sure those feature maps are reliable in the target domain? Are there situations in which the method might be stuck in local optima or even diverge? Conducting an ablation study on the pre-rendered feature maps would be valuable.

We discuss this exact problem in Section 3.2 (Multi-Pose Initialization) and provide an ablative example in Figure 7. The source model does sometimes get stuck in local optima when evaluated on the target domain data, and therefore the estimated result may depend on the pose initialization. Therefore, we do multi-pose initializations for our method, and use the final estimates to update only robust, local vertex features irrespective of the global object pose estimated. We have added an ablation for multi-pose initializations in our updated draft (Table 4).

Q3. In Sec.3.2.1, the $\delta$ parameter seems crucial for effectively updating vertex features. The authors mentioned they chose a $\delta$ such that the majority of source domain features lie within the likelihood score. How to set in practice? Is it constant? Intuitively, as the vertex features are updated, one would expect the similarity between vertex features and image features to increase. Does it make more sense to update $\delta$ accordingly?

For our experiments, we empirically find that a normalized value between 0.85-0.9 for the parameter works sufficiently well for our model and approximates the coverage of the majority of source domain vertex features. Tightening the parameter during model updates helps the model learn faster. We thank the reviewer for their suggestion! Here is the ablative experiment for bus category on OOD-CV dataset with different threshold values:

Q4. It seems the method is time-consuming due to the iterations and pre-rendering. However, the actual time consumption during testing is unclear.

Thanks for pointing out that this should have been discussed in the paper. Evaluation time for our model (if not doing batched inference) is ~0.33 seconds/sample on a Nvidia RTX 2060 GPU. We have added it in the paper.

We thank the reviewer for their time.

W1. It is recommended to evaluate the proposed method on commonly used datasets for category-level pose estimation, such as REAL275 or Wild6D.

• Our focus in this work is largely on outdoor setups (e.g. OOD-CV) due to relatively more diverse OOD conditions (weather, lighting, context, etc.) available in such datasets.

• Both WILD6D+ and REAL275 datasets are indoor RGBD datasets in NOCS [1] format, which are used primarily for robotic applications and have 5-6 categories. Historically, works have either focused on datasets like Pascal3D+, ObjectNet3D+ or REAL275, WILD6D+ and we follow the same convention. This is likely due to incompatibility between the camera parameters and data formats of the two kinds of datasets. It is not trivial to convert one to another (or convert models of one kind to another) and is beyond the scope of this review period due to technical and time-related limitations. We believe that combining these two baseline branches would strengthen the research field, but it remains a future work.

• Most models [3,4,5] using these datasets rely on the depth channel to run PnP or combine depth with NOCS to solve pose so they cannot be trivially modified as an RGB model (to compare with our method) and require extensive work.

• We therefore follow the convention of datasets used in previous works related to our pose estimation method.

As an alternative for the Synthetic to Real evaluation done in REAL275 and WILD6D+ benchmarks, here are the results for synthetic to real dataset using the SyntheticP3D source domain dataset[2], which consists of 6 synthetic objects. The real dataset is Pascal3D+.

[1] Wang et al. - Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.

[2] Yang et al. - Robust Category-Level 3D Pose Estimation from Synthetic Data.

[3] Lee et al. - TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation.

[4] Zhang et al. - Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild.

[5] Liu et al. - IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation.

W2. It would be beneficial to include relevant works [3,4,5,6] in the paper to provide a comprehensive overview of the existing literature in the field.

Thanks for pointing out the works. We have added the mentioned references in the updated draft.

We thank the reviewer for their valuable feedback. In response to their comments:

W1a. Poor presentation. I think this paper holds good insights and corresponding technological contributions. Yet, the presentation of the whole paper is relatively poor, making it hard to follow the overall message. Figure placement is not well-aligned with the text context. Figure 3 seems to be a very important description of explaining one of the main contributions of this paper to match sub-vertices, while there is no mention in the main manuscript referring this.

We regret that the presentation of our initial draft was deemed poor by the reviewer. According to their suggestions, we have updated our paper. We have reframed the main content to make it easier to follow, improved the quality and placement of figures and made sure that all figures are properly referenced in the text. We request the reviewer to take a look at the newer version to see if the presentation issues are addressed.

W1b: Moreover, the main manuscript refers to figures in appendix very often, which is very inconvenient to read, while some of these figures seem important enough to be contained in the main paper for effective description. (ex, Figure 5) I believe that ablation studies regarding several design choices of the proposed method should be contained in the main manuscript as well, since they can effectively validate the authors claim.

With our restructuring of the paper, we have moved some important details up from the appendix to the main text as we agree with the reviewer that some of them are too important to be deferred to the appendix. We have updated old Figure 3 and moved old Figure 5 to the main draft (refer to Figures 1 and 3 in the updated draft). Unfortunately, numerous experimental analysis performed in this work can not all fit in the space constraints of the main text. We hope that the current restructuring in the new draft bolsters the contributions that we made with this work. We are happy to take more suggestions regarding the same and address any more issues that the reviewer thinks still exist.

Q1a. Baseline methods and other datasets Except OOD-CV results, only previous baseline is NeMo. Can the authors explain why there can't be other methods in this comparison? Being better than only one baseline is not enough to claim the superiority of the proposed method.

We thank the reviewer for drawing attention to this point. We have added results for other baselines' extended results tables in our updated draft (Table 6 and Table 7 (page 22 and 23)) which includes four more comparative methods for every experiment. As can be gleaned from these baselines, like OOD-CV, our method performs exceedingly well even when dealing with extreme UDA setups.

Q1b. Also, while I acknowledge that the paper mainly focuses on data corruption scenarios, is it possible to compare this type of approach in conventional category-level object pose benchmarks like CAMERA and REAL275 datasets provided by NOCS? I believe it would strengthen the authors' motivation if it can be generally applied to conventional syn-to-real UDA scenarios.

Thank you for drawing our attention to CAMERA and REAL275 datasets.

• Our focus in this work is largely on real, outdoor setups (e.g. OOD-CV) due to relatively more diverse OOD conditions (weather, lighting, context, etc.) available in such datasets.

• Conventionally, 3D pose estimation works have either focused on setups and datasets like PASCAL3D+, ObjectNet3D or on more robotic application based synthetic to real datasets like REAL275, CAMERA and WILD6D+. This is due to some distinct technical and usage difference between these dataset setups. It’s non-trivial to convert the camera calibrations, annotations and dataformats from one to another and would require a significant amount of time and is likely out of scope for the review period. To the best of our knowledge, there is no recent work which has used both of these kinds of datasets together, and we believe this would be a good future work to combine these two types of datasets into one format.

• Both CAMERA and REAL275 datasets are indoor RGBD datasets, which are used primarily for robotic applications and have 5-6 categories. Most models[3,4,5] using these datasets[1] rely on the depth channel to run PnP or combine depth with NOCS[1] to solve pose so they cannot be trivially modified as an RGB model (to be compared with our method) and require extensive work.

• As recommended by the reviewer - for a conventional synthetic to real analysis, here are results for image only unsupervised domain adaptation from SyntheticP3D source domain dataset[2] (which has 6 synthetic object classes) to real Pascal3D+ target domain dataset.

[1] Wang et al. - Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.

[2] Yang et al. - Robust Category-Level 3D Pose Estimation from Synthetic Data.

[3] Lee et al. - TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation.

[4] Zhang et al. - Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild.

[5] Liu et al. - IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation.

Q2. GT reliability Ground truth pose illustrated in Figure 1 and 5 seems to be not perfectly aligned with the image. Can the authors explain how these GTs are obtained, and how they are utilized? Are they only used for evaluation?

We want to clarify that the CAD objects in old Figure 1 and 5 (updated Figure 3) are for visualisation only. We use Shapenet3D models and rotate them according to the dataset’s annotations (GT) and model’s (Ours, NeMo) pose estimation results. There might be some noise in GT pose annotation as well the visualization setup which might be the reason for non-perfect alignment. The images as well as their ground truth annotations (GTs) in the figures are obtained from the OOD-CV dataset.

Dear Reviewer,

We have updated our paper's presentation according to your comments and have covered all the technical questions that you've asked. It'll be of vastly beneficial to us if you can tell us what additional improvements you'd like to see in our paper, so we can make them before the discussion period ends. If you are satisfied with our method and believe we have covered your technical questions, it'll be reassuring if you can kindly update our paper's rating to reflect the same.

We thank the reviewer for their valuable feedback. In response to their comments:

W1. As mentioned in the article, an observation is that global information is noisy, but some local details are robust. I hope there is rigorous explanation and quantitative analysis here to support this hypothesis.

Section 3.3 provides theoretical justification of how local robustness helps the domain adaptation problem. To see whether this local robustness is observed in real datasets, we calculated the percentage of correctly detected vertices in the target domain. In Figure 1 (in the updated draft), we see that this value has a minimum of 5-20% robust vertex features in the object category “Car”. After adaptation using our method, this percentage increases significantly. This is further illustrated for other object categories in the appendix (Figures 5, 9, 10). If some specific quantitative analysis is missing, we request the reviewer to point it out so that we can update it.

W2. Ablation is not enough, (e.g., Ablation on top-n

It is unclear what the reviewer means by missing ablation studies (top-n) in our 3-D pose estimation problem. We believe that we have ablated all major design choices and method motivations in Section A.5. In summary, our ablation (Section A5) includes ablation of

a. Our motivation and observations regarding local part robustness (Figure 5, 9, 10),

b. The efficacy of the Selective Vertex Feature Update,

c. False Positives in vertex-Feature Similarity,

d. Effect of the learnable concentration parameter,

e. Multi-Pose Initialization,

f. Quality of vertex Features after Selective Adaptation.

In addition, we have added the following more ablations in the revised version:

g. Effect of number of clutter features hyperparameter,

h. Additional ablation figures to prove the local part robustness observation mentioned in the previous response paragraph. (Figures 9 and 10)

If the reviewer is referring to the top-n multipose initializations, we do observe faster convergence with increasing number of initial poses. However, during multipose initialization, we only consider estimated poses whose rotational matrix distance (refer to Section 4 (Metrics paragraph)) from one another is at least $\pi/5$. If the estimated poses are less than the threshold for all, we only consider the pose with the highest similarity to the image feature vector. We did not see any significant improvement in accuracy using more than 5 initial poses for our experiments. Here are ablation results for the car category for different numbers of pose initializations when adapted on OOD-CV (Combined dataset).

If there are specific ablation studies that the reviewer feels are missing, we would be happy to run them.

W3. I think this method is similar to the iterative optimization often used in instance-level post-processing. It is unfair to compare this method with other single forward methods.

It will be helpful if the reviewer can clarify which methods or provide citations of the works that use iterative optimization and are similar to our UDA method. To the best of our knowledge, our method is the first method to do source-free, image-only unsupervised domain adaptation for 3D object pose estimation. If the reviewer refers to the render-and-compare optimization used in our model, we would like to clarify that this is indeed a common optimization method used for a number of pose estimation methods such as [a,b,c] (this is not our contribution) and it is standard practice to compare such methods with feedforward methods as can be seen in these works.

[a] Li et al. - DeepIM: Deep Iterative Matching for 6D Pose Estimation

[b] Wang et al. - NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

[c] Chen et al. - Category level object pose estimation via neural analysis-by-synthesis

W4. I'm not sure if it's right to say "normalized real-valued feature activations" ? ( The fourth line from the bottom of the third page)

The phrase "normalized real-valued feature activations" refers to feature activations of a neural model which are real numbers and are normalized to have unit norm. We request the reviewer to be more specific as to why they believe the usage is incorrect.

W5. The figures in this article are very rough and difficult to understand.

We have updated the visual clarity of our figures in the main draft (refer to Figures 1, 2 and 3) and modified them to be easier to understand. We hope that the revised figures address this concern of the reviewer.

Dear Reviewer,

As the reviewer-author period comes to an end, we are yet to receive any response or acknowledgment from you regarding our response to your initial review. We have also asked some clarifications regarding your initial review so we can frame our response better. Please let us know if you have any further questions or if our clarifications and paper updates are satisfactory for an improved rating.

We are happy that reviewers - who have engaged in a conversation with us - have acknowledged that we have addressed all their concerns and questions. We find their positive outlook towards our work sincerely encouraging.

For the reviewer (xesL) who hasn't been able to engage with us in a conversation, we've ensured that all your concerns have also been addressed through our comprehensive response.