Assembly Fuzzy Representation on Hypergraph for Open-Set 3D Object Retrieval

1. Visualization (Weakness 1) We apologize for the lack of sufficient visualization results. We provide some visualized examples of the retrieval results in Fig. R3 of the rebuttal PDF and we will provide more.

2. Equations (Weakness 2) We have revised the expression and explanation of these two equations. Eq 2. is the goal of the open-set retrieval task (minimize the expected risk), which is a widely accepted type for task definition [1-3] in the retrieval field. Eq 4. is the definition of HIConv layer, which followed the commonly used type for graph-based convolution [4-6].

   [1] View-based 3-D object retrieval[M]. Morgan Kaufmann, 2014.
   [2] Hypergraph-based multi-modal representation for open-set 3D object retrieval. TPAMI, 2023.
   [3] SHREC’22 track: Open-set 3D object retrieval. 2022.
   [4] How powerful are graph neural networks? ICLR, 2019.
   [5] Hypergraph neural networks. AAAI, 2019.
   [6] Hgnn+: General hypergraph neural networks. IEEE TPAMI, 2022.

3. Framework (Weakness 3): Thanks for your valuable suggestion, We apologize for the typos in Figure 2, the "inter-object" should be corrected as "intra-object". We have restructured the presentation of the proposed framework, which consists of two sequentially connected modules: IAE and SFR. a) The IAE module takes basic part features (as explained in Answer 4) as input. This module employs a structure-aware convolution layer and a set of auto-encoders to achieve assembly fusion of different parts within an object. In this module, the structure-aware convolution layer is implemented by constructing an isomorphism hypergraph and a hypergraph isomorphism convolution function (as explained for Eq. 4 in Answer 2).
   b) The SFR module takes assembly embedding for each object as input, utilizing structure-aware feature smoothing and distillation through hypergraph convolution and memory bank reconstruction, respectively. Finally, this module generates the final features (fuzzy embeddings) for similar object matching based on feature distance, thereby enabling retrieval.

4. Input (Weakness 4) Thanks for your valuable suggestion. The inputs for our framework are the part features rather than dense point clouds, and we do not need ground truth 3D part segmentation as input (as described in lines 259-263 and lines 476-478 of the submission). Instead of extracting features from the segmented parts of the point cloud, we use a segmentation network to obtain point-wise features for each point and then average these point-wise features to obtain part features. As shown in Fig. R2 of the rebuttal PDF, the steps are as follows:

   a) Input the point clouds of an object.
   b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network.
   c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework.

   We do not need dense point clouds nor the segmentation ground truth for input. As shown in Tab. R1 of rebuttal, compared with the SOTA point cloud segmentation methods [7][8], the features obtained using PointNet did not significantly affect the results. Therefore, we believe our method will not collapse if the segmentation is not perfect. Besides, our framework is a feature-driven method (as described in lines 157-158) and does not directly process raw data (dense point clouds). Thus, its robustness to SE(3) transformations and partial back-projection of point clouds is equivalent to that of the point feature extraction network. These backbone networks have rotation equivariant and adaptability to partial data, as proven in famous works such as [7-11]. Therefore, we believe the method will not collapse when confronted with SE(3) transformations and partial data.
   [7] Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NIPS, 2022.
   [8] Segment any point cloud sequences by distilling vision foundation models. NIPS, 2024.
   [9] Pointnet: Deep learning on point sets for 3d classification and segmentation. CVPR, 2017.
   [10] Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NIPS, 2017.
   [11] Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.

5. Limitations (Limitations) We have provided a brief discussion in line 330-332.

   a) Experiments of various parts. We only conduct limited experiments on the influence of varying parts in the manuscript. We conducted more experiments with the number of input part features set to 3 ($n=2$) and 5 ($n=4$). As shown in Tab. R1 of the rebuttal PDF, both the 4-part (ours) and 5-part settings show performance improvements over 3-part, indicating that more detailed segmentation provides richer information for assembly-based retrieval. However, our 4-part setting currently exhibits better than 5-part. We can infer that once the number of parts reaches a certain level, the utilization of part information also becomes saturated.

   b) Discussions of societal impacts. As shown in line 158-159 and Appendix C Algorithm 1, the proposed HARF framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data and such as e.g. text, audio, video, and their pieces. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets and code immediately after the anonymous review period.

Dear Reviewer,

We would greatly appreciate any updates or feedback you might have regarding our responses to your initial comments. Your insights are valuable to us as we work to improve our paper.

If you need any additional information or clarification from our side, please don't hesitate to let us know.

Thank you for your time and consideration.

We sincerely appreciate your positive feedback and professional comments on our work. Your valuable suggestions have been crucial in improving the quality of our paper. We also appreciate the rating improvement, and will carefully revise the manuscript according to your review comments and ensure the rigor of the experimental results and references.

Looking forward to academic discussions with you after the anonymous period of NeurIPS 24, if possible! We are willing to share all our experiences, datasets, and codes of this work.

Thanks for your rebuttal. I will increase my rating to borderline accept

Response for Reviewer ZtnM

We sincerely thank you for the valuable comments and advice, which provided important guidance for us to enhance the rigor and coherence of our paper and directed the focus of our future work.

1. About the generalization ability of the model (Answer for Weakness 1 and Weakness 3):
   Thanks for your valuable suggestion. As an early exploration of open-set learning in a fine-grained way, we believe this paper should focus on exploring the new assembly-based retrieval task and designing a novel open-set learning paradigm based on both inter-object and intra-object correlations. Therefore, we selected the three 3D object datasets with typical geometrical structures for experiments, which is representative of a fine-grained way for an open-set environment. Experimental results demonstrate the necessity and effectiveness of the assembly-based paradigm and framework. However, further improving the generalization ability of our framework and extending it to encompass more intertwined factors in complex open-set environments is one of the key directions for future work. Specifically, we have preliminarily experimented with an adaptive Scale isomorphism Computation method for generalization across datasets and domains, inspired by [1] and [2]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of parts and labels inspired by [3].
   [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024.
   [2] Zhou J, et al. Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.
   [3] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.

2. About module-wise computational requirements (Answer for Weakness 2):
   Our experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.

3. About more ablation studies (Answer for Weakness 4):
   Thanks for your valuable suggestion. We conducted more ablation studies, especially on HIConv and other neural layers. Our method employs only a single layer of HIConv, therefore, we explored the impact of more layers of HIConv. Besides, we replaced the HGNN with KAN network [4] in the SFR module. As shown in Tab. R1 of the rebuttal PDF. The full version of our framework yields the best performance, these results indicate the effectiveness of our design in assembly-based open-set 3D object retrieval.

4. About the public datasets (Answer for Weakness 5):
   Thanks for your valuable suggestion. The proposed datasets are constructed based on public datasets without open-set settings, making them suitable for open-set retrieval experiments. As mentioned in Answer 1, we will continue to explore assembly-based retrieval research in our extension version, by constructing more datasets and conducting more experiments.

5. About the significance of the proposed method (Answer for Questions):
   Thanks for your valuable suggestion. We provide statistics of our framework (HAFR) and the second-best method (HGM$^2$R). As shown in Tab. R3 of the rebuttal PDF, statistical information proves the significance of the results.

6. About failures cases and limitations (Answer for Limitations):
   Thanks for your valuable suggestion. We provide some failure cases in Fig. R3 of the rebuttal PDF. In these failure cases, the query objects (rocket, pistol) and the wrong-matched target objects (car, motorbike) share a certain similarity in their part segmentation. Although the significant performance improvement of the HARF framework demonstrates the necessity of assembly-based research in open-set learning, these cornet cases also indicate the necessity of the equilibrium between different levels of labels, which needs the balance of global semantic and local geometry information. This issue is the same as the generalization ability mentioned in Weakness 1. However, this paper focuses more on the fundamental challenge brought by part assembly for open-set retrieval. Therefore, we only consider typical segmentations in this study. As mentioned in Answer 2 above, we are currently conducting research to address these more complex environments. Thank you for your keen observations and academic insights.

Thank you again for your valuable suggestions, especially your professional advice on future work in assembly-based open-set learning.

1. Varying numbers of parts (Weakness 1)
   HAFR takes 4 part features as input for each object in this manuscript for now. As shown in Fig. R2 of the rebuttal PDF, the steps for part features generation are as follows:
   a) Input the point clouds of an object.
   b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network.
   c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework ( $n=3$ for this paper).
   We conducted more experiments with the number of input part features set to 3 ($n=2$) and 5 ($n=4$). As shown in Tab. R1 of the rebuttal PDF, both the 4-part (ours) and 5-part settings show certain performance improvements over the 3-part, indicating that more detailed segmentation provides richer information for assembly-based retrieval. However, our 4-part setting currently exhibits the best performance (better than the 5-part). We can infer that once the number of parts reaches a certain level, the utilization of part information also becomes saturated.
2. Computational complexity and efficiency (Weakness 2)
   Our experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF. The SFR module occupies a very small parameter space (less than 3%) and is directly affected by the size of datasets. The IAE module only consumes up to 34%, which determines the effectiveness with high-dimensional features. We believe that our method can remain effective when scaling to larger datasets or higher-dimensional part features.
3. PartNet (Weakness 3)
   The ShapeNet Part [1] dataset we used and the PartNet [2] dataset are both fine-grained annotated subsets of ShapeNet, with PartNet having a greater variance in the number of parts per object. As an early exploration of open-set learning in a fine-grained way, we believe this paper should focus on exploring the new assembly-based retrieval task and designing a novel open-set learning paradigm based on both inter-object and intra-object correlations. Therefore, we selected the ShapeNet Part with more evenly distributed parts rather than PartNet, which can reflect the core challenges of a fine-grained approach in an open-set environment. We have preliminarily experimented with an adaptive Scale isomorphism computation method for better generalization, inspired by [3] and [4]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of parts and labels inspired by [5] for the future work.
   [1] A scalable active framework for region annotation in 3d shape collections[J]. ACM ToG, 2016.
   [2] Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding[C]. CVPR, 2019.
   [3] Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024. [4] Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.
   [5] Hypergraph dynamic system[C]. ICLR, 2024.
4. SDML on OP-COSEG (Weakness 4)
   SDML proposes a scalable multimodal learning paradigm for retrieval by predefining a common subspace, aiming to minimize the intra-class difference. In the context of assembly-based retrieval, the SDML method projects all parts into a class-related common space to achieve unification. However, this approach results in an overall feature shift, leading to the loss of unique geometrical information of parts and the correlations among them. As shown in Fig. R3 of the rebuttal PDF, we can find that compared to the OP-SHNP and OP-INTRA datasets, the objects in the OP-COSEG dataset exhibit stronger symmetry, leading to less influence of this biased unification. Furthermore, NDCG is a metric that considers the global perspective, thus SDML achieves a slight advantage (0.11%) of NDCG on OP-COSEG through this globally biased unification. However, the significantly worse results on both other datasets and other commonly use metric demonstrate the limitation of this method.
5. Reproducibility and open access (Weakness 5 and 6)
   We have provided a brief description of implementation details and dataset generation in the Appendix. Besides, we are well prepared and will release the datasets, code, configurations, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We are willing to share our experiences on this (OpenReview) or other open-source platforms.
6. Failure cases (Weakness 7)
   We provide some failure cases in Fig. R4 of the rebuttal PDF. In these failure cases, the query objects (rocket, pistol) and the wrong-matched target objects (car, motorbike) share a certain similarity in their part segmentation. Although the significant performance improvement of the HARF framework demonstrates the necessity of assembly-based research in open-set learning, these cornet cases also indicate the necessity of the equilibrium between different levels of labels, which needs the balance of global semantic and local geometry information. This issue is the same as the generalization ability mentioned in Weakness 1. However, this paper focuses more on the fundamental challenge brought by part assembly for open-set retrieval. Therefore, we only consider typical segmentations in this study. As mentioned in Answers 1 and 3 above, we are currently conducting research to address these more complex environments.
7. Writing (Limitations):
   We apologize for the typos and writing issues in this manuscript. We will conduct a thorough review and revision of the entire paper to ensure the clarity and rigor.

1. Generalization ability and comparison (Weakness 1)
   All categories of the testing set are unseen during training (widely accepted of open-set retrieval [1-2]), the retrieval results in this paper are experimented on the unseen categories. The compared results between different methods are conducted under the same settings, and training weights.
   We have explained this open-set setting in Section 5.1 (lines 254-255) and Appendix A (Table 4). We will provide a more detailed explanation in the revised version of this paper. Quantitative and qualitative results show that HARF achieves significant improvements over existing methods. Under this open-set setting, this improvement sufficiently demonstrates the superiority of HARF in terms of generalization capability.
   [1] Open-environment machine learning. National Science Review, 2022.
   [2] Hypergraph-based multi-modal representation for open-set 3d object retrieval. TPAMI, 2023.
2. Related works (Weakness 2)
   In our submission, We have provided a brief review of closed-set 3D object retrieval methods in lines 78-88, and a summary of recent open-set 3D object retrieval methods in lines 95-97.
3. More comparisons (Weakness 3)
   We provide more comparison between modern methods[3][4] for open-set 3d object retrieval. As shown in Tab. R1 of the rebuttal PDF, the results indicate that these two methods have similar performance to the existing multimodal method (HGM$^2$R). Our HAFR framework shows significant improvements over all these SOTA open-set retrieval methods. This improvement highlights the limitations of existing retrieval paradigms in open-set environments and demonstrates the superiority of our assembly-based open-set retrieval paradigm.
   [3] Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.
   [4] Openshape: Scaling up 3d shape representation towards open-world understanding. NIPS, 2023.
4. The approach for finding similar objects (Weakness 4 and Question 1)
   After obtaining the fuzzy embeddings (feature vectors) for all objects, we follow the common distance-based approach in the multimedia retrieval field to find similar objects, which is a widely accepted practice in recent decades [5-7]. Given a feature vector (fuzzy embedding) of query object: a) Calculate the Euclidean distance between the query feature vector and all target object feature vectors.
   b) Sort these distances in ascending order to get the top-$n$ nearest objects. c) Determine whether the class labels of the top-$n$ nearest objects are the same as the query object label and calculate metrics such as mAP, NDCG, ANMRR, and PR-Curve, where $n$ denote the hyper-parameter of the evaluation metrics and can be chosen based on the specific scenario.
   [5] A survey of content-based image retrieval with high-level semantics. PR, 2007.
   [6] 3-D object retrieval and recognition with hypergraph analysis. TIP, 2012.
   [7] Triplet-center loss for multi-view 3d object retrieval. CVPR, 2018.
5. More ablation on the Memory Bank (For Question 2)
   The memory bank we used has 128 anchors and 512 dimensions. The memory bank is a commonly used knowledge distillation method in deep learning. Specifically, it constructs several anchors (feature vectors) and then learns the activation scores of the target embeddings relative to all anchors. The memory bank is independent of the classification layer, and its size usually does not affect the performance of the network when the number of categories changes. This has been validated in methods of multiple fields[8]. We have conducted more ablation studies on the memory bank. As shown in Tab. R1 of the rebuttal PDF, changes in the memory size have almost no impact. We believe HARF remains effective even if the dataset contains more than 16 categories.
   [8] Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. CVPR, 2021.
6. Methods for part feature extraction (For Question 3)
   We have adopted a neural network model for part segmentation to obtain part features (as described in lines 259-263 and lines 476-478 of the submission). However, instead of extracting features from the segmented parts of the point cloud, we use a segmentation network to obtain point-wise features for each point and then average these point-wise features to obtain part features. As shown in Fig. R2 of the rebuttal PDF, the steps are as follows:
   a) Input the point clouds of an object. b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network. c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework. Where $n$ is the hyper-parameters in this paper.
   Therefore, in our framework, we do not need to know how many parts each object should be segmented into. We have conducted more comparisons on the segmentation method. Compared with the SOTA point cloud segmentation methods [9][10], the features obtained using PointNet did not significantly affect the results. Therefore, we believe the method remains applicable in real-world scenarios and is not influenced by the choice of the segmentation network.
   [9] Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NIPS, 2022. [10] Segment any point cloud sequences by distilling vision foundation models. NIPS, 2024.
7. Undefined concepts (Weakness 4)
   We apologize for the lack of sufficient visualization results. The isomorphism loss mentioned in line 189 is a typo, and it should mean the loss function for the IAE module (section 4.2.3). The integration function of line 189 is an averaged function for multiple features followed [2].

Response for Reviewer 3mhr

We sincerely thank you for the valuable comments and advice, which provided important guidance for us to enhance the rigor and coherence of our paper and directed the focus of our future work.

1. About the ablation study on $K$-value (Answer for Weakness 1): We further conduct ablation studies on hyper-parameters $K$ to validate the influence of uncertainty hyperedge in the leverage hypergraph. As shown in Fig. R1 of the rebuttal PDF, as the $K$ varies, the performance of the proposed method remains stable and outperforms the compared method in most ranges. However, the performance ceases to improve and may even slightly decrease after reaching a certain value, indicating that the extraction and utilization of high-order correlations have reached saturation. Besides, we can find that different datasets have different peak values of $K$. A larger K-value indicates the requirements for more extensive and deeper capturing of high-order correlations. Your suggestions have inspired our future work, and we will focus on balancing the performance and complexity of the structure-aware network to achieve optimal relationship modeling. We will also include this part of the experiment and analysis in the paper.

2. About the unique contributions of IAE (Answer for Weakness 2): The IAE module is designed to obtain assembly embeddings from multiple part features. Compared to the multimodal embedding method in HGM²R, which generates averaged embeddings of different modalities from a global perspective, the IAE module aims at assembled embeddings with geometric-semantic consistency for all parts from a global-local collaborative perspective. Specifically:

   a) The IAE module constructs a structure (the Isomorphism Hypergraph) to capture geometric correlations, such as the order and quantity among the input parts, and then utilizes them for structure-aware fusion. However, HGM²R only uses an averaged-guided fusion method with simple auto-encoders.

   b) Guided by the isomorphism hypergraph, the IAE module designs the Hypergraph Isomorphism Convolution (HIConv) layer that combines geometric and semantic information to generate embeddings collaboratively. However, HGM²R uses a naive MLP for feature mapping and fusion, which loses the high-order information within the input.

   Although the HGM²R designs a satisfactory approach for multimodal embedding, its semantic-only averaged embedding paradigm is not suitable for part assembly, which requires the collaborative use of information from different domains. Based on HGM²R, the IAE module of our method introduces the Isomorphism Hypergraph and Hypergraph Isomorphism Convolution to achieve part assembly with geometric-semantic consistency. Experimental results in both the paper and the rebuttal PDF demonstrate the necessity and superiority of the IAE module.

3. About the improvement on OP-COSEG (Answer for the last Weakness): Although HGM²R uses a hypergraph structure, they only construct an inter-object hypergraph at the feature smoothing stage (stage 2), without considering the intra-object correlations between different parts of an object, which directly determine the accuracy of the embeddings. Our method constructs hypergraphs for capturing both intra-object and inter-object correlations, as mentioned in the Answer 2 above. We provide examples of the three datasets in Fig. R4, we can find that compared to the OP-SHNP and OP-INTRA datasets, the objects in the OP-COSEG dataset exhibit stronger symmetry, meaning that the isomorphism between parts is more significant. Ignoring geometric information during assembly embedding may lead to inconsistencies within the same category. These challenges result in only slight improvements over non-hypergraph methods. However, our method tackles the challenge of assembly isomorphism and unification, achieving part assembly with geometric-semantic consistency, leading to significant performance improvements on the OP-COSEG dataset.

4. About societal impacts (Answer for Limitations) As shown in line 158-159 and Appendix C Algorithm 1, the proposed HARF framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data and such as e.g. text, audio, video, and their pieces. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets, code, configs, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.

Thank you again for your valuable suggestions, especially your professional advice on future work in assembly-based open-set learning.