3D Object Representation Learning for Robust Classification and Pose estimation

We thank the reviewer for the constructive feedback.

My main concern with this paper is lack of novelty, in particular with respect to: "Neural Textured Deformable Meshes for Robust Analysis-by-Syntheses" by Wang et al. (which is cited in the submission). This paper seems to follow the same procedure, using mostly the same techniques (feature extraction, 3D mesh representation, contrastive divergence to make 3D features more invariant, etc.). There seem to be some differences (e.g. it seems the submission the 2D to 3D matches is done differently, and the submission, unlike Wang et al., allows to do classification without the more computationally expensive step of 3D pose estimation, but I am not certain), but these differences are not explained in the submission, nor seem significant enough to warrant sufficient novelty. Moreover, Wang et al. achieve very comparable results, and in some cases superior performance (e.g. for L3) on Occluded PASCAL3D+ (Compare Table 1 in Wang et al. to Table 1 in the submission). I would have expected a much more detailed comparison to the most relevant papers (in particular "Neural Textured Deformable Meshes for Robust Analysis-by-Syntheses" by Wang et al.), with clear articulation of what the differences are, and pros and cons of the proposed method with respect to these alternatives

We are sorry for the confusion with the related paper [G]. We cited the paper for the sake of completeness, and should have better pointed out that it is contemporaneous concurrent work that was not published and only available on ArXiv at the time of submission. We note that according to ICLR reviewer guidelines we should therefore not be required to compare it as a baseline in our experiments (see last answer in the FAQ in [F]). Nevertheless, when comparing our method with [G] the reviewer must have misread the results of the Tables in [G] since our method outperforms [G] significantly at classification and pose estimation in all IID and OOD scenarios. In addition, as the reviewer requested, we contacted the authors and received their code to test on the corrupted-P3D+ dataset. We report the results in Table 4, and can observe that our model outperforms all types of corruption with a large margin, and we are around 20 percent higher overall.

We further want to emphasize that, our proposed method performs classification efficiently in real-time, which takes on average 0.02 seconds per image, whereas [G] simply runs a brute force search during inference which takes on average 0.5 seconds per image on the Pascal3D+ dataset under the same setup, suffering from a 25 times slower inference in addition to the lower accuracy and robustness, and hence making it practically infeasible.

The way how we formulate the contrastive loss is also different, which can be illustrated by the loss function in two papers. We use a vMF-based contrastive loss that encourages features to be distinct through a dot product operation whereas [G] uses a Euclidean-distance based loss estimating the distance between feature vectors. The main focus of [G] is different from this paper. Its goal is to obtain deformable (exact) geometry of the object in its forwarding. In summary, these two papers have major differences in both low-level implementation and high-level goals, and performance [G] in terms of both accuracy and efficiency for the classification task.

I also think some claims are not adequately supported by the results, for example "exceptionally robust classification and pose estimation results" as claimed in the abstract

We will tone down this paragraph in the abstract. However, we note that our proposed model significantly outperforms the best baselines at image classification in OOD scenarios, e.g. +5.8% over ConvNext on occluded-P3D+, +21% over Swin-T on OOD-CV, and +3.7% over ViT-16 on corrupted-Pascal3D+. We also note that the ranking among the baselines varies significantly across datasets, e.g. Swin-T achieves the best ranking among baselines on OOD-CV but only ranks third on corrupted-Pascal3D+. In contrast, our proposed method consistently outperforms the baselines across all OOD scenarios.

Table 4: Classification accuracy results on corrupted-PASCAL3D+ under 12 different types of common corruptions. Our approach outperforms DMNT by a large margin for all corruptions.

[F] Reviewer guide: https://iclr.cc/Conferences/2024/ReviewerGuide

[G] Angtian Wang, Wufei Ma, Alan Yuille, and Adam Kortylewski. Neural textured deformable meshes for robust analysis-by-synthesis. arXiv preprint arXiv:2306.00118, 2023

We thank the reviewer for the constructive feedback.

More detailed discussions and comparisons with NeMo. NeMo is highly related to the proposed paper, but the discussion is missing in the introduction and related works. As far as I can see, the 3D object representation in the paper is already proposed by NeMo. NeMo already finds that using a cubic for an object class can achieve good performance in 3D pose estimation. They also use contrastive loss for training and render-and-compare strategy for pose estimation as in this paper. I think the difference is that the paper extends this existing 3D representation to training on multiple categories and uses it for classification. Based on this point, I think a huge part of the contributions of this paper belong to NeMo, and the real contributions are limited.

Our method indeed builds on several previous works while substantially extending them. Compared to NeMo, we make several key contributions: (1) An architecture with a class-agnostic backbone, while NeMo uses a separate backbone for each object. This enables us to introduce a class-contrastive loss that enables object classification, while NeMo is not capable of classifying images. As illustrated in the t-SNE plots in reviewer-wnbh’s answer. (2) A principled way of speeding up the inference without having to perform render-and-compare, which advances the brute-force testing and optimization of other concurrent methods [G] (see answer to reviewer-lehm for more details). (3) A comprehensive mathematical formulation that derives a vMF-based contrastive training loss that is different from Euclidean-distance based contrastive loss of NeMo. (4) We demonstrate the possibility to exploit individual vertices during inference rather than considering vertices collectively as a mesh in NeMo, opening potentials tackling segmentation or object part detection tasks. Finally, all our advances lead to (5) substantial improvements in terms of OOD robustness over all baselines at image classification, while at the same time performing on par with models that were specifically designed for robust pose estimation, such as NeMo.

For the evaluation of classification, I think the paper should compare with the state-of-the-art 2D object detectors for classification to show the advantages. For example, I think the YOLO can be easily trained using the same training data as the proposed method, i.e., the projected 2D bounding boxes from the 3D cubics.

Thank you for the suggestion. We trained YOLO using the official repository D on the Pascal3D+ data with standard data augmentation in terms of scale, rotation and mirroring data augmentation. From the table below, we can observe that YOLO obtains similar performances for IID data, but suffers even more than the other classification baselines under out-of-distribution shifts in terms of occlusion, whereas our proposed model achieves much stronger robustness. While refining our YOLO results through additional investigation might be possible, we speculate that utilizing the official training code for YOLO might lead to potential overfitting, given its original focus on detection rather than classification. We anticipate that more advanced data augmentation (such as CutOut or CutPaste) techniques could enhance generalization in YOLO's performance. Notably, it is crucial to emphasize that our method, in contrast, does not necessitate data augmentation and demonstrates effective generalization.

Table 3: Classification results on P3D+ and occluded-P3D+ comparing Yolo and our approach.

We thank the reviewer for the constructive feedback.

All typos should be checked and corrected to improve writing quality (e.g., in the first paragraph of the Introduction Section, "… gradient-based optimization on a specific training set (Figure ??).").

We are sorry for the typos and will revise the manuscript thoroughly.

There is a lack of an efficiency comparison with existing methods. It seems that the high efficiency in classification is a critical contribution of this work. A comparison of the inference speed and the number of parameters between the proposed framework and other methods can further support this claim.

In terms of classification, our method achieves similar real-time performance as the other baselines, processing over 50 images per second. Among the baselines there is a large variation in terms of the number of parameters (Resnet: 84M, Swin-T: 28M, ViT-b-16: 86M), but we do not observe any correlation with OOD robustness. Compared to those, our model contains 83M which is comparable. However, concerning the pose estimation baseline Nemo, we have a much lower parameter count (83M vs 996M) and faster inference speed. We will make sure to emphasize this more in the revised paper.

We thank the reviewer for the constructive feedback.

Authors start the paper with the statement: “we pioneer a framework for 3D object representation..”, the statement is in contrast with various papers that investigate object representation using techniques on how to map image features to 3D models of faces and vertices. For example M1 or M2.

We acknowledge the relevance of these papers and their techniques in mapping image features to 3D models. We will revise our claim in a more nuanced manner to better align with the state of the art. Additionally, we will incorporate references to M1 and M2 in our related work section, recognizing their significance in the exploration of object representation techniques.

$\kappa$ (eq. 3) is fixed to constant, how is the parameter fixed? Is it from validation data, guess, tuning on testing data?

Originally, we did not explore estimating the concentration parameter $\kappa$ because there is no closed form solution to it. Therefore, we set it to $\kappa=1$. Estimating these parameters is non-trivial because of the lack of a closed-form solution and potential imbalances among the visibilities of different vertices. Nevertheless, motivated by the comment of Reviewer-wnbh, we now tested the effect of approximating the parameter using the method as proposed in [A]. In the Figure 1, we can observe that the concentration of the learned features is slightly higher, where the object variability is low (e.g. wheels, bottle body, …), whereas it is lower for objects with very variable appearance or shape (e.g. airplanes, chairs, sofa, …). When integrating the learned concentration parameter into the classification and pose estimation inference we observe almost no effect on the results (see Table 1). This needs to be studied more thoroughly, but our hypothesis is that the learned representation compensates for the mismatch of the cuboid to the object shape and therefore the weighting of the concentration parameter does not have a noticeable effect.

Table 1: Classification results on P3D+ and OCC- P3D+ comparing our results by including and excluding the concentration parameters from the classification during inference.

Do the baselines use 3D? (It looks that authors use 3D but baselines don't).

To the best of our knowledge, our method is the first in the literature to use a 3D representation for image classification. Moreover, it is not clear how to provide existing architectures with additional 3D annotation. Based on the feedback of reviewer-wnbh, we conducted an experiment where we extended several of the classification baselines with an additional pose estimation head. In particular, we follow the popular approach of casting the pose estimation problem as bin classification by discretizing the pose space [C]. We train the models with both the image classification and pose estimation loss until convergence. As can be observed from the table below, the additional 3D information does not improve the classification performance. Moreover, this claim is supported by the t-SNE plots shown in Figure 2. Simply adding 3D-pose information via an additional branch does not induce the network to learn any 3D-aware structure in the representation, whereas the 3D-aware structure is directly built-in in our approach, thus leading to large benefits in terms of robustness and interpretability.

If the reviewer has any alternative baselines to suggest that leverage the advantages of 3D, we are open to incorporating and testing them in our evaluation.

Table 2: Classification results on P3D+ and occluded-P3D+ comparing resnet baseline performances by training a network per task and a single network for both. We don’t observe any synergy in the unique model trained on both tasks.

[A] Sra, S. (2011). "A short note on parameter approximation for von Mises-Fisher distributions: And a fast implementation of I_s(x)". Computational Statistics. 27: 177–190

[B] Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang. Starmap for category-agnostic keypoint and viewpoint estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 318–334, 2018.

We are pleased that our efforts to provide clarification have successfully addressed all concerns of all reviewers that were raised throughout the discussion period. We have meticulously integrated all feedback from the reviewers already in the revised manuscript.

We express our sincere gratitude for the thoughtful consideration and constructive feedback provided during the discussion process.