Scaling 3D Compositional Models for Robust Classification and Pose Estimation

1. I'm concerned about the limited improvement over the baselines, given that the model is much more complicated than the ViT/ResNet baseline. For example, as shown in Tab 4, pose acc=57.6 for proposed method vs 56.9 for ViT on L0 occlusion, and 16.0 vs 15.7 for L3. One advantage is training time reduction, but I find this less important than the inference speed, as training only need to be done once.
2. I'm not sure if the comparison is fair, as some improtant experiment details are missing: L376-383 describes how virtual corrption on data is applied. Is this corrpution only applied to test set or also the training set? If only applied to test set, what data augmentation is used during training? If applied also to training, then this is different from the augmentation used to train baselines (we apply standard data augmentation (i.e., scale, rotation, and flipping) during training, quoted from L419-420), and the comparsion is not fair. The fairness of comparison is especially important given that the improvement is very small. Please clarify.
3. Fig 1 is misleading: the is actually not as big as shown in the figure. The authors should show the ticks on each axis.
4. False claim #1: In abstact and intro, the authors mention heavily their method can scale to higher number of vertices in the mesh. However, the main contribution of this paper (Dynamically Weighted Compositional Contrastive Learning) solves mainly the scaling to increasing number of classes. Neither is the generalization to higher number of vertices supported by any experiment. Therefore, I don't think this is a valid point.
5. False claim #2: L457 the authors say "...about the drastic decrease both in memory usage...by our model in Table 2.". However, there is no numbers for memory usage in Tab 2. Conceptually I also don't think the proposed model will have advantage in peak memory consumption, as the proposed dynamically weighted graph is pruned gradually, so dense connection is required at the beginning of the training.
6. False claim #3: L151 the authors claim "We have a compositional structure of objects and parts and so we can use contrastive learning on the parts". However, the object is represented as cubiods as a whole and no parts are used in their method.
7. I'm concerned about the real-world application of this method. The authors show that the method can genealize from real-world data to the diffusion model generated synthetic image, but it would be more convincing if the authors do the opposite, i.e., sim to real. Moreover, the pose estimation task is 3D instead of 6D. That means the object has to be at a fixed scale and centered at the location of the image. Also, it can only estimate pose of a fixed class of objects, since classification happens before pose estimation, which seems more limited than the stadard 6D pose estimation setup. The authors should at least present some qualitative results on in the wild images.
8. Conceptually, why is the proposed model more robust to occlusions for pose estimation? In Eqn 7, the authors maximize the feature similarity of both foreground region (defined by the rendered object mask) and background region. This foreground mask doesn't model occlusion, therefore the occluder (which has image feature corresponding to background) will be matched with object features. I think this will cause confusion in the pose etimation process. Some further clarification and discussion are desired.
9. L292 mentions there is an ablation for subsampling the vertices in contrastive loss but I don't find the table. Which exact ablation experiment should I look at?
10. The paper doesn't discuss about the failure cases or limitations. Also, more visualizations are desired to better understand how the leanred object representation looks like and how the pose estimation is performed.

Some minor problems that won't affect my rating: M.1. Eqn 7, from my understanding this term should be maximized since you want to have a high matching score. In this case I would not call it a loss, as in ML loss is usually supposed to be minimized. Maybe name it "energy" instead. M.2. Some abbreviations used without definition, and are inconsistent. E.g., two forms of IID used ("i.i.d" in L69 and IID in L103) without definition. O(n^2) is used in many places, but n is not defined.

1. There is still room for improvement in the writing of this paper. For me, it takes some time to understand the logic in the method section, especially since it contains a lot of unofficial expressions. Take a few examples:

a) line 269: w.r.t 1. from distanced vertex features of the same object 2. vertex features of other object classes and 3. background features. （The three points listed lack punctuation, and I am unsure if this listing format is formal.）

b) line 286: If we try to trivially scale this loss to |Y | classes, we find that the number of contrastive terms scales approximately by a quadratic (n2) factor! (Informal)

2. There are also some unclear parts in the writing. For example, how the confusion matrix is computed from the calibration data split (line 293) and how the candidate poses set which is used to select the optimal initial pose α are predefined (line 366).

• The paper does not show results on the Omni3D [A] benchmark: a dataset comprised of 6 datasets. It would be good to quantitatively compare agains the Cube R-CNN [A] baseline on the pose estimation task on this task. Since Omni3D does not have its own pose estimation metric, you could use your metric to compare against the Cube R-CNN baseline on the pose estimation sub-task.

• What is the inference time of this method compared to Cube R-CNN baseline?

• Why does the method learn good neural volumes even if the number of input images is only one. Neural volumes are poorly constrained for 1 input view in my experience.

Reference:

• [A] Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild, Brazil et al, CVPR 2023

The assumption of the independence of feature vectors at each mesh vertex is a strong assumption. The authors need to come up with a strong justification for this assumption especially if one considers the spatial coherence of mesh vertices, i.e., the vertices that are in close spatial proximity would be expected to have similar feature vectors. Also, is the contrastive learning technique presented in the paper adequate to enforce the independence of feature vectors at distinct mesh vertices? Also, it is not clear how the training would scale with updates to the classes? Would the training need to be performed from scratch with the addition of a new class?