Coarse-to-Fine 3D Part Assembly via Semantic Super-Parts and Symmetry-Aware Pose Estimation

Q1: Evaluate the model's performance on unseen category.

In the manuscript, we adopt the experimental settings of DGL [1], RGL [2], Score-PA [3], IET [4], and 3DHPA [5] to evaluate models on the three largest categories of PartNet, where our model achieves superior performance compared to the baselines.

To evaluate the performance of our method on unseen category, we test with the model trained on Chair category to predict assemblies for shapes in Lamp category. As shown in Table R4-1, our model (Our Chair⟹Lamp) outperformed the baseline methods (* Chair⟹Lamp, where * represents the compared models) on 4 out of 5 metrics (SCD, PA, CA, QDS), achieving second-best performance on WQDS.

These results demonstrate the generalization ability of our approach, which leverages semantic guidance to learn part-level structures and produce coherent and accurate assemblies even for unseen category.

Table R4-1. Performance of models trained on Chair for assembly prediction on unseen Lamp category

Q2: Report computational cost of the proposed components.

To evaluate the computational cost of our proposed components, we report key metrics for a single forward pass with a batch size of 64, including the number of parameters (Million, M), GPU memory usage (GB), forward time (ms), and GFLOPS. We analyze the CFPA model and its variants, each excluding specific components as: 1)-2) CFPA-w/o-refine/coarse MHA that exclude the multi-head attention in the pose refinement/coarse pose estimation stage; 3) CFPA-w/o-OT that removes the OT-based super-part construction; 4)-5) CFPA-w/o-SRFP/LRFP that remove the short-/long-range feature propagation; 6) CFPA-w/o-MP that removes the message passing in long-range feature propagation; 7) CFPA-w/o-CA that removes the cross-stage attention; 8) CFPA-w/o-SL that removes the symmetry-aware loss. We also report the computational cost of the baseline method that removes all of the above components.

As shown in Table R4-2, removing these components leads to a reduction in computational cost compared to the full CFPA model. Notably, some of these components are specifically designed to operate based on the relationships between parts and super-parts rather than directly on point clouds, with the maximum number of parts limited to 20 and super-parts to 16. This design inherently requires only a small computational cost. Additionally, certain components, such as OT, short-range feature propagation, and message passing, do not introduce any learnable parameters, further contributing to their relatively low computational cost.

Table R4-2. Computational cost comparisons

Q3: It will be interesting to add visualization or analysis of learned super-parts or their semantic consistency across shapes.

While figures are not permitted in this rebuttal, we visualize the correspondence for the basic parts with their nearest super-parts, identified by the minimal cost in the optimal transport map $T$. These visualization reveal several key tendencies:

A. Parts with similar functions are mapped to the same super-part, showcasing strong semantic coherence. For instance, chair legs are typically grouped into one super-part, while chair backs and seats are often mapped into another super-part, reflecting their interchangeable functional roles.

B. Unique or uncommon parts, such as decorative elements or isolated structural pieces, tend to be assigned to specific super-part.

C. The learned super-parts demonstrate semantic consistency across shapes. For instance, despite variations in chair designs (e.g., chairs with armrests versus those without), parts with similar functions, such as chair legs and seat backs, are consistently grouped into their respective super-parts, while armrests are mapped into separate super-parts.

These findings highlight the model's ability to capture semantic and functional relationships across shapes, while effectively handling structural variations and unique components.

Q4: It will be interesting to evaluate the generalization of the proposed symmetry-aware loss on other baselines.

To evaluate the generalization ability of the proposed symmetry-aware loss, we conducted experiments by integrating it into various baseline methods. As shown in Table R4-3, the models equipped with our symmetry-aware loss (* -w/-SL, where * represents the baselines) achieve improved performance.

These results demonstrate that the symmetry-aware loss is not only effective within our framework but also generalizable across diverse baseline methods. This further validates its robustness and applicability in 3D part assembly tasks.

Table R4-3. Comparison of different baselines with our symmetry-aware loss

Q5: Lack comparison with Imagine in Tab.1.

We will include a comparison with Imagine [6] in the revised manuscript and clarify the distinctions between the two methods. While both tackle 3D part assembly tasks, Imagine utilizes structural guidance from 2D images, making it a conditional approach. In contrast, our method addresses the task in an unconditional setting. Despite these differences, our method outperforms Imagine in terms of the PA metric on the Chair (69.24% vs 65.88%) and Table (68.48% vs 65.13%) categories.

We sincerely thank you for your thoughtful feedback and valuable suggestions. We will incorporate the additional analysis and experiments into our revised manuscript.

[1] Zhan, Guanqi, et al. Generative 3d part assembly via dynamic graph learning. NeurIPS 2020.

[2] Narayan, et al. Rgl-net: a recurrent graph learning framework for progressive part assembly. WACV 2022.

[3] Cheng, Junfeng, et al. Score-pa: score-based 3d part assembly. BMVC 2023

[4] Zhang, Rufeng, et al. 3d part assembly generation with instance encoded transformer. IEEE Rob. Autom. Lett., 2022, 7(4): 9051-9058.

[5] Du, Bi'an, et al. Generative 3d part assembly via part-whole-hierarchy message passing. CVPR 2024.

[6] Wang, Weihao, et al. Imagine: image-guided 3d part assembly with structure knowledge graph. AAAI 2025.

Q1: How the semantic super part $\\{h\_j\\}\_{j=1}\^M$ comes from $\\{f_i\\}_{i=1}^N$? What is the definition of geometrically similar parts in line 159?

(1) Semantic super-parts.

The semantic super-parts $\\{h\_j\\}\_{j=1}\^M$ are computed as weighted aggregation of basic part features $\\{f_i\\}_{i=1}^N$, where the weights are set as transport map $T$, as formulated in Eq. (1) of the manuscript.

The transport map (weight) $T$ is obtained through an entropy-regularized optimal transport (OT) process, which minimizes the objective function described in Eq. (2) of the manuscript. This optimization is efficiently solved using Sinkhorn's algorithm [1], which iteratively updates the transport map and super-parts until convergence.

These semantic super-parts act as middle-level representations and serve as feature prototypes, offering meaningful structural abstractions that facilitate subsequent feature propagation and pose prediction.

(2) Geometrically similar parts.

We follow DGL [2] and compare the axis-aligned bounding box sizes of each part. Two parts are considered geometrically similar if the absolute difference in their bounding box sizes is below a predefined threshold (set to 0.1). For example, under this criterion, parts such as the legs of a table or the seat and backrest of a chair can be identified as geometrically similar if their bounding box size differences satisfy the threshold. This definition for geometrically similar parts is also provided in Appendix A.

[1] Distances, Cuturi M. Sinkhorn. Lightspeed computation of optimal transport. NeurIPS 2013.

[2] Zhan, Guanqi, et al. Generative 3d part assembly via dynamic graph learning. NeurIPS 2020.

Q2: Ablation study should contain baseline method. More ablation experiments are encouraged.

We present the results of our 1) baseline method and the full CFPA model in Table R3-1. For the baseline, we systematically remove key components of our approach, including semantic super-parts, dual-range feature propagation, cross-stage attention, and the symmetry-aware loss.

To further evaluate the designs in our model, we conduct additional ablation studies as follows.

A. To demonstrate the effectiveness of entropy-regularized OT for semantic super-part construction, we conduct experiments with: 2) CFPA-w/-Att.map: This variant replaces our transport map computed by entropy-regularized OT with an attention-based transport map as used in [3]. 3) CFPA-w/-$L_2$-dis: This variant replaces the cost function in OT with $L_2$ distance in feature space. The results in Table R3-1 show that these variants achieve lower performance than our full CFPA model, demonstrating the effectiveness of entropy-regularized OT and inner product as cost function for semantic super-part construction.

B. To evaluate the effectiveness of our symmetry-aware loss for handling self-symmetric and geometrically similar parts, we conduct experiments with: 4)-6) CFPA-w/-SL in [4]/[5]/[6]: These variants replace our symmetry-aware loss $\mathcal{L}_{sym}$ in Eq.(15) with the symmetry-handling losses proposed in [4]/[5]/[6]. The results in Table R3-1 show that CFPA with our symmetry-aware loss achieves better performance compared to these symmetry-handling losses, demonstrating its ability to effectively handle self-symmetric and geometrically similar parts.

C. To demonstrate the effectiveness of our symmetry-aware loss compared to methods that explicitly use encourage distinct placement losses for geometrically similar parts, we conduct ablation studies with: 7) CFPA-w/-SL in [7]: This variant replaces our symmetry-aware loss $\mathcal{L}\_{sym}$ in Eq.(15) with the loss function proposed in [7], which encourages distinct placement for geometrically similar parts. 8) CFPA-w/-SL-$L_2$: This variant replaces our symmetry-aware loss $\mathcal{L}\_{sym}$ in Eq.(15) with $L_2$ distance applied within geometrically similar parts. The results in Table R3-1 show that CFPA with our symmetry-aware loss outperforms these variants, highlighting its superiority in handling geometrically similar parts than explicitly enforcing distinct placement.

We hope that these comparisons, along with the variants analyzed in the ablation study of the manuscript, provide a thorough and convincing evaluation of our method.

Table R3-1. Ablation studies for CFPA

[3] Xin Wei, et al. Learning generalizable part-based feature representation for 3d point clouds. NeurIPS 2022.

[4] Zhang, Jiahao, et al. Manual-PA: Learning 3D Part Assembly from Instruction Diagrams. arXiv:2411.18011

[5] Zhang, Rufeng, et al. 3d part assembly generation with instance encoded transformer. IEEE Rob. Autom. Lett., 2022, 7(4): 9051-9058.

[6] Li, Yichen, et al. Learning 3d part assembly from a single image. ECCV 2020.

[7] Zhang, Ruiyuan, et al. Scalable geometric fracture assembly via co-creation space among assemblers. AAAI 2024.

Q3: The method part needs more explanation about how this module solves the issues.

We adopts a coarse-to-fine strategy for 3D part assembly, which face challenges including the lack of explicit semantic structure, difficulties in integrating semantic information, inconsistencies between different prediction stage and ambiguities caused by self-symmetry or geometric similarity. Below, we clarify how each module specifically resolves these issues:

A. To address the lack of explicit semantic structure, we propose semantic super-parts as high-level structural abstractions. These are dynamically constructed via optimal transport (OT) (Eqs.(1-2)), avoiding reliance on rigid hierarchies. By grouping semantically similar parts, this module provides a semantic abstraction for guiding coarse pose estimation. This mechanism equips the framework with explicit, data-driven semantic structure, thereby improving global structural understanding and the overall quality of assembly.

B. To guide localized feature learning with the learned semantic abstraction, we introduce a dual-range feature propagation mechanism. Locally, it captures fine-grained semantic relations between parts and their nearest super-parts (Eq.(3)); globally, it aggregates holistic context via attention (Eq.(4)). This design ensures that local feature learning is effectively guided by multi-scale high-level semantic structure, enabling precise part placement while maintaining global structural coherence.

C. To promote consistency between different stages and transfer global information from the coarse to the refinement stage, we introduce a cross-stage attention module. By fusing complementary cues across stages, this module enables the refinement process to be guided by global structural context, thereby reducing inconsistencies and improving overall assembly coherence.

D. To resolve ambiguities from self-symmetry and geometric similarity, we propose a symmetry-aware loss that supervises pose prediction with explicit self-symmetry and geometric similarity modeling (Eq.(13), Eq.(14)). This allows the network to tolerate multiple valid configurations for symmetric or interchangeable parts. By doing so, our method accommodates symmetry-induced ambiguities and supports diverse yet valid assembly solutions.

E. Beyond the above designs, to enhance feature interaction and discrimination, we employ multi-head attention in both the coarse and refinement stages (Eq.(7), Eq.(9)), facilitating effective information exchange among parts. Furthermore, message passing among geometrically similar parts (Eq.(5)) encourages feature consistency for interchangeable components, while an instance encoding strategy preserves unique part representations and prevents prediction collapse.

We sincerely thank you for your feedback and suggestions. We will incorporate the additional analysis and experiments into our revised manuscript.

Q1: Why are only 3 categories of shapes shown in Table 1?

We followed the experimental settings of DGL [1], RGL [2], Score-PA [3], IET [4], and 3DHPA [5], which evaluate models on the three largest categories of PartNet. As shown in Table 1 of the manuscript, our CFPA consistently achieves higher overall performance compared to the baseline methods.

In addition, we conducted experiments on the Storage category, with comparative results coppyed from SPAFormer [6]. As illustrated in Table R2-1, our method outperforms the baseline methods on the SCD and PA metrics, while achieving the second highest performance on the CA metric. Together with the results in Table 1, these results highlight the effectiveness of our model for accurate part assembly.

Furthermore, we performed few-shot transfer learning experiments by training our model on the Chair category and fine-tuning it using only 10 shapes from the Table category. As shown in Table R2-2, our fine-tuned model (Ours Chair$\rightarrow$Table) outperforms both other fine-tuned models (* Chair$\rightarrow$Table, where * represents competing methods) and our model trained from scratch (Ours trained from scratch). These results underscore the effectiveness of our approach under limited training data and demonstrate its superior generalization capabilities.

Table R2-1. Performance comparison of different methods on Storage

Table R2-2. Performances of different methods for few-shot transfer learning

[1] Zhan, Guanqi, et al. Generative 3d part assembly via dynamic graph learning. NeurIPS 2020.

[2] Narayan, et al. Rgl-net: A recurrent graph learning framework for progressive part assembly. WACV 2022.

[3] Cheng, Junfeng, et al. Score-pa: Score-based 3d part assembly. BMVC 2023

[4] Zhang, Rufeng, et al. 3d part assembly generation with instance encoded transformer. IEEE Rob. Autom. Lett., 2022, 7(4): 9051-9058.

[5] Du, Bi'an, et al. Generative 3d part assembly via part-whole-hierarchy message passing. CVPR 2024.

[6] Xu, Boshen, et al. SPAFormer: Sequential 3D Part Assembly with Transformers. 3DV 2025.

Q2: The qualitative results demonstrated in Figure 4 feels a bit underwhelming.

Unconditional part assembly is inherently challenging due to the complexity of understanding assembly patterns and accurately placing symmetric or similar parts. Unlike conditional methods, this task operates without predefined guidance, requiring the model to learn intricate relationships between parts.

From a qualitative perspective, our model demonstrates better connectivity and more reasonable handling of symmetric or similar parts. For example, in the Chair category, it predicts more accurately aligned legs with stronger connectivity. In the Table category, wheels are placed more reasonably, and for Lamps, lampshades are positioned more effectively. These results highlight the robustness of our approach in capturing meaningful part relationships and producing plausible assemblies.

Quantitatively, our model consistently outperforms baseline methods across multiple metrics as detailed in experiment results of the manuscript. These numerical results further validate the effectiveness of our approach in achieving accurate and robust part assemblies.

Q3: How do you account for multiple possible assemblies for the same set of parts? How to specify the desired final assembly through conditioning?

We explicitly accounts for multiple possible assemblies. As in Eq.(14), we consider diverse configurations of similar parts to be valid and constrain the prediction of each part to be consistent with the ground truth of its similar counterparts. This ensures structural plausibility while accommodating the inherent ambiguity of symmetric or geometrically similar parts.

To further address potential collapse caused by identical predictions for similar parts, we incorporate part instance encoding during the refinement stage. This mechanism distinguishes individual parts and avoids over-simplification. Additionally, we employ a shape loss (Eq.(12)) to enforce global structural consistency, ensuring that the overall assembly remains coherent while preserving diversity.

3D part assembly with conditions, such as images or textual descriptions, offers a promising approach to achieve the desired final configuration. For example, establishing 2D-3D correspondences through projection methods can effectively constrain the uniqueness of 3D assemblies. Additionally, leveraging deterministic relational links between parts in images to guide the modeling of inter-part relationships in 3D space is a worthwhile direction for further investigation.

Currently, we focus on unconditional part assembly, where predictions are made without external guidance. We greatly appreciate your suggestion, as integrating conditional information to enable unique and deterministic predictions represents a highly meaningful and valuable avenue for future exploration.

We sincerely thank you for your thoughtful feedback and valuable suggestions. We will incorporate the additional analysis and experiments into our revised manuscript.

Q1: Physical meaning & intuitive interpretation of super-part from OT.

Physically, as Eq.(1), each super-part is a weighted aggregation of basic part features learned via optimal transport, forming compact prototypes that encode multiple related parts. Intuitively, these super-parts group functionally or geometrically similar parts into semantic clusters, providing mid-level abstractions without predefined hierarchies.

Visualization: Figures are not allowed in rebuttal. We visualize the correspondence between parts and their nearest super-parts identified by transport map $T$. Figures show: 1) parts with similar functions (e.g., chair legs, backs & seats) tend to cluster within the same super-part; 2) unique or uncommon parts (e.g., decorative elements) are tend to be assigned to one specific super-part; 3) the super-parts exhibit semantic consistency across shapes, consistently grouping functionally similar parts.

Q2: Difference of super-parts from global [1] or GNN-based aggregation [2]. Motivation & advantage of intermediate super-parts.

Difference: Unlike global feature aggregation [1] that compresses part information into single vector and loses intermediate structures, our super-parts preserve mid-level semantics by clustering parts into coherent groups. Compared to GNN-based method [2] that model inter-part relations via message passing but may suffer from over-smoothing, we build hierarchical representations to encode structural relationships, bridging local details and global organization.

Motivation & advantage: Object parts often exhibit similarities in appearance or functionality, naturally forming semantic groups. These relationships provide a foundation for coarse-to-fine approach in part assembly, where intermediate representations help progressively refine local details into coherent structure. This hierarchical design enhances semantic consistency across parts and guides precise part assembly, as validated by ablation study 1)-3) in manuscript.

[1] Wu, R., et al. Leveraging se(3) equivariance for learning 3d geometric shape assembly.

[2] Scarpellini, G., et al. Diffassemble: a unified graph-diffusion model for 2d and 3d reassembly.

Q3: Why use entropy-regularized OT & inner product.

OT offers a flexible, probabilistic alignment of parts to super-parts, enabling soft associations compared to rigid clustering or geometry-based grouping. Entropy regularization ensures smooth assignments, avoids sparsity, and enables efficient Sinkhorn-based computation.

Replacing OT with k-means or geometric grouping (ablation 2)-3) in manuscript), or replacing entropy regularization with attention-based map as in [3] (CFPA-w/-Att.-map in Table R1-1), results in worse performance, highlighting effectiveness of entropy-regularized OT.

Inner product measures compatibility between part and super-part by capturing directional similarity in feature space. Its linearity ensures computational efficiency and stability during training.

Replacing inner product with $L_2$ distance-based cost function (CFPA-w/-$L_2$-dis. in Table R1-1) resulted in worse performance, validating its suitability for this task.

Table R1-1. Results on Chair

[3] Xin W., et al. Learning generalizable part-based feature representation for 3d point clouds.

Q4: Physical meaning & intuitive interpretation of dual-range feature propagation & cross-stage attention.

Physically, Dual-range feature propagation allows each part to capture both global semantic structure (Eqs.(3-4)) and local geometric details (Eq.(5)), bridging global and local contexts. Cross-stage attention leverages coarse-stage semantic cues to refine part features, enabling more accurate pose estimation and consistent assembly.

Intuitively, Dual-range feature propagation enables each part to integrate local details with global structure, while cross-stage attention ensures feature refinement maintains semantic consistency across stages.

Q5: Analysis/validation on symmetry-aware loss vs other symmetry-handling methods.

Analysis: IET [5] use Chamfer Distance (CD) and instance encoding to address self-symmetry and distinguish similar parts, Li et al. [6] and Manual-PA [4] use Hungarian algorithm for one-to-one matching, with Manual-PA further introducing InfoNCE loss to reduce matching uncertainty. In contrast, we minimizes CD over all flipped variants (Eq.(13)) to explicitly model multiple symmetric solutions and avoid bias from uneven point sampling. For geometrically similar parts, we relax strict matching (Eq.(14)) to allow interchangeable roles, enabling structurally equivalent assemblies.

Validation: Replacing symmetry-handling loss in Eq.(15) with alternatives from [4–6] (CFPA-w/-SL in [4]/[5]/[6] in Table R1-2) improves performance over baseline without symmetry handling (CFPA-w/o-SL). CFPA with our symmetry-aware loss achieves best results, demonstrating the benefit of explicitly modeling symmetric configurations.

Table R1-2. Results with symmetry-handling losses

[4] Zhang, J., et al. Manual-PA: Learning 3D Part Assembly from Instruction Diagrams.

[5] Zhang, R., et al. 3d part assembly generation with instance encoded transformer.

[6] Li, Y., et al. Learning 3d part assembly from a single image.

Q6: Are 8 reflections sufficient? How reflection outside SE(3) affects chirality preservation? Computational cost for multiple forward passes and potential strategies.

Sufficiency of the eight axis-aligned reflections is guaranteed because all input parts are pre-aligned using a chirality-preserving PCA method as in DGL [7], before being fed into the model.

Chirality is preserved during both training and inference: 1) For inference, input parts are pre-aligned using a chirality-preserving PCA as in DGL [7], and the model predicts rotations with quaternion, ensuring no chirality inversion occurs. (2) For training, although Eq.(13) formulates the reflection-based symmetry constraint as applying reflections to input, in practice, for computational efficiency, we instead apply these reflections only to ground-truth poses of self-symmetric parts.

Computational cost: In implementation, the potential computational overhead for symmetry-aware supervision is significantly reduced by applying axis-aligned reflections only to ground-truth poses of self-symmetric parts, rather than reflecting input parts. As in Table R1-4 (last two rows), using symmetry-aware loss does not increase parameter count but leads to increased GPU memory (26.60GB→27.04GB) and forward time (802.72ms→806.80ms).

Other strategies: 1）designing symmetry-equivariant architectures to ensure consistent outputs under input reflections; 2）grouping symmetric variants by aggregating losses or sampling representatives; 3）employing hierarchical processing to first capture global symmetries before refining local variations.

[7] Zhan, G., et al. Generative 3d part assembly via dynamic graph learning.

Q7: Why not encourage distinct placement for collapse?

By minimizing prediction differences to the most similar ground-truth counterpart (Eq.(14)), we accommodate multiple plausible part configurations induced by geometric similarity. The part-wise instance encoding and shape loss (Eq.(12)) further maintain coherent global arrangements without collapsing similar parts into identical poses.

As in Table R1-3., replacing symmetry-aware loss $\mathcal{L}_{sym}$ in Eq.(15) with distinct-placement constraint as [8] (CFPA-w/-SL in [8]) or using $L_2$ loss (CFPA-w/-SL-$L_2$) on predictions of geometrically similar parts results in worse performance.

Table R1-3. Results with constraints for geometric similar part

[8] Zhang, R., et al. Scalable geometric fracture assembly via co-creation space among assemblers.

Q8: Computational cost of modules.

Table R1-4. presents computational cost of CFPA and its variants without key modules of semantic super-parts via OT (CFPA-w/o-OT), short/long-range feature propagation (CFPA-w/o-SRFP/LRFP), message passing (CFPA-w/o-MP), cross-stage attention (CFPA-w/o-CA), and symmetry-aware loss (CFPA-w/o-SL). The baseline model refers to CFPA without any of these modules. '#Para.(M)' is number of parameters(in millions). All variants are tested on Chair with batchsize=64, forward time is measured for one iteration.

These modules operate on part or super-part levels rather than the point cloud, and the overhead introduced by these components in the full CFPA remains manageable and does not significantly impact scalability.

Table R1-4. Computational cost

We will incorporate the analysis on model and experiments, and correct the typo.