• The paper lacks some key experiments that would help assess the significance of the method:
  • Necessity of render-and-compare: What would be the difference if, during inference, the approach did not rely on render-and-compare optimization but instead directly relied on feature matching between the image and the neural template mesh to obtain 2D part segmentation? How would this approach compare to the inference pipeline proposed in the paper?
  • Alternative approaches based on 2D-3D matching (or canonical mapping): Several methods (e.g., [1, 2, 3]) could also achieve part estimation for images in the same category from a single annotated mesh by label transfer through matching. One of these methods should be considered as a baseline for 2D part estimation.
• The proposed method achieves part segmentation with one-shot annotation by integrating existing methods; however, the problem formulation itself lacks significance. Technically, it overlaps heavily with category-level object pose estimation. For instance, [4] already demonstrates the effectiveness of combining NeMo and 3D-DST for object pose estimation, while [5] applies deformable template meshes to object pose estimation. Simply adapting these techniques for part annotation does not appear sufficiently significant.
• The inference process of the method relies on optimizing multiple parameters at the part level (pose, scale, shape latent), and shape latent training uses only 8 instances per category for instance-level training. This raises concerns about the robustness of the optimization process.

[1] Canonical Surface Mapping via Geometric Cycle Consistency

[2] SELF-SUPERVISED GEOMETRIC CORRESPONDENCE FOR CATEGORY-LEVEL 6D OBJECT POSE ESTIMATION IN THE WILD

[3] To The Point: Correspondence-driven monocular 3D category reconstruction

[4] GENERATING IMAGES WITH 3D ANNOTATIONS USING DIFFUSION MODELS

[5] Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis

Major

• The notation and writing are confusing and difficult to follow.

  • L193: Should $z_k$ be $z_y$? Is it a per-mesh shape latent feature rather than per-vertex? Additionally, Fig. 3 and L295 suggest that each part shape may have an independent shape latent.
  • L188 and L234: Is $U_k$ in L188 the same as $\theta_j$ in L234?
  • The notation lacks clarity for the learnable parameter used for the image encoder $\psi$, creating confusion, as it seems the vertex deformation network $\phi$ is not learnable.
  • L228: $\phi$ is reused as an image encoder, although it was previously defined as the mesh deformation network in L193.
  • L243: Notation for $j$ and $i$ is missing.
  • L238: Why is the camera pose represented as $\mathbb{R}^3$? Is it only for translation? Why is rotation ignored and not represented as $SE(3)$?
  • In Eq. 3, what is $\mathcal{y}$?
  • L251: The background term is not defined. Is it Eq. 3 of [Bai et al., 2023]?
  • L269: Does $W$ represent trained weights?
  • Figure 2: The figure and notation in the manuscript do not match, making the pipeline hard to understand. The trainable modules, such as the mesh deformation network (L193), vertex feature encoder (L205), and image encoder (L228), should be clearly indicated.

• A comparison with foundation-based models would be preferable. Foundation models exhibit part-level segmentation capabilities in a one-shot setting (e.g., Fig. 3 of [1]). Additionally, previous work has leveraged this capability for category-level, part-level 3D shape reconstruction using only 2D images [2]. Since these approaches do not require per-part annotation nor 3D models with or without annotation, additional experiments/explanations for comparison would be beneficial to clarify the advantage of the proposed method.

Minor

• The idea of a neural mesh with "render and compare" (learning a surface embedding for the template mesh and corresponding per-pixel feature encoder for image-aligned shape reconstruction) is similar to [3,4] but is not referenced. Although these methods require a pretrained DensePose model [3] or are purely optimization-based [4], the manuscript should at least discuss the differences from these methods.

References

1. Liu et al. "MATCHER: Segment Anything with One Shot Using All-Purpose Feature Matching." ICLR 2024.
2. Yao et al. "Hi-LASSIE: High-Fidelity Articulated Shape and Skeleton Discovery from Sparse Image Ensemble." CVPR 2023.
3. Kulkarni et al. "Articulation-aware Canonical Surface Mapping." CVPR 2020.
4. Yang et al. "ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction." NeurIPS 2021.

• The proposed approach requires 3D datasets of aligned meshes for each category of interest. Further, a separate model needs to be trained for each category. This limits the scalability and applications as it requires both aligned 3D data and many models. It also makes it difficult to extend to deformable or articulated objects.
• Evaluations are only done on four categories (car, aeroplane, bicycle and boat). Is it possible to use datasets like PACO (CVPR23) or PartNet-E? It's difficult to gauge the significance or true capability of Part321 unless larger and more complex datasets are used.
• Baselines are 2D segmenters trained on small labeled datasets which is likely suboptimal for those methods. It is important to include comparisons to off-the-shelf methods such as Matcher (ICLR24) which are off the-shelf-techniques that can be directly applied to 2D part segmentation. This is not a one-to-one comparison, but is an important datapoint to understand the significance of what Part321 has achieved in an absolute sense.

Minor

• L151 - "specially accuracy" typo
• L248 - $\theta$ is not defined as a variable

• The authors state in line 108 that a single 3D annotation is used for each class, yet they mention in line 323 that 'During training, we use 30 CAD models … for each category.' This creates ambiguity around the training setup. Does this imply that out of the 30 CAD models, only one is annotated, or are additional annotations being used in some form?

• The evaluation metric used in this paper appears misaligned with the task objectives. While the primary focus of the paper is on 3D part recognition, instead of using 3D segmentation metric, the authors rely on detection metrics for evaluation, which may not effectively capture segmentation performance.

• The experiments on 2D part segmentation in Table 1 are limited in scope. The paper only compares against earlier work, without including recent methods (e.g., [1]) specifically designed for 2D part segmentation.

• In the 3D part segmentation experiments, the authors do not include comparisons with any baseline methods (e.g., [2,3]) that address similar tasks. While training settings may differ slightly, such comparisons are essential to contextualize the performance and validate the effectiveness of the proposed approach.

• The experiments appear limited to only a few categories (e.g., Car, Aeroplane, Bicycle, Boat). I believe results on more categories should be included to show the effectiveness of the proposed method.

[1] Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles, CVPR'22

[2] PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models, CVPR'23

[3] 3x2: 3D Object Part Segmentation by 2D Semantic Correspondences, ECCV'24

The robustness of the proposed technique to occlusions and other image degradations has not been discussed or studied. For example, it is not clear how robust the proposed technique is to partial occlusion of an object part in an image. This would apply to both, the training phase where the correspondences are established and the testing phase where the object parts are recognized. The authors to test their proposed technique on synthetically occluded images and images with added degradations (such as noise and blur), both during training and testing phases. The extent of occlusion and image degradation should be systematically varied in terms of extent and intensity respectively and the performance of the proposed technique quantified and noted accordingly.