Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph

Response for Reviewer vaz9

We sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.

1. About the writing and symbols (Answer for Weakness 1):
   We apologize for these typos. We will conduct a thorough review and revision of the entire paper to ensure the clarity and rigor of the writing.
   1.1 (For Weakness 1.1) The typos $\mathbf{F}^k$ in line 152 and $\mathcal{T}$ in line 155 should be correctly written as $f^k$ and $\mathcal{X}$, which denote the basic features of $k$-th modality and aggregation function, respectively.
   1.2 (For Weakness 1.2) We have restructured the data flow in the HRE module for each object:
   $\{ f^k_i \}^M_{k=1} \rightarrow \mathcal{A}_m \rightarrow \mathcal{X} \rightarrow c_i$
   $(c_i+e_i) \rightarrow \Psi \rightarrow r_i \rightarrow \Phi \rightarrow \hat{m}_i$
   where $e_i$ is an encoding of the coarse label, which is a learnable vector of the same dimension as $c_i$. Specifically, retrace encoding is implemented by nn.Parameter for each coarse category, objects within the same coarse label share the same retracing encoding. During computation, it is element-wise added to $c_i$ and then input to $\mathcal{A}_r$. $\hat{m}_i$ is used solely for loss calculation in the HRE module, using coarse labels for supervision to guide the accuracy of the retrace embedding representation. It does not participate in the computations of the SET module.
   Based on your suggestions, we have revised the pipeline diagram and added more symbols, as shown in Fig. R2 of the rebuttal PDF.
   1.3 (For Weakness 1.3) We removed the presentation of space in lines 157-163 to enhance the readability of the paper. Specifically, we revise this paragraph as follows:
   ...then $\mathcal{A}_r$ compresses the unified embedding $c_i$ aligned with $e_i$ into retrace embedding $r_i$ and does the reverse reconstruction $\hat{m}_i$ for supervision...

2. About the traditional 3D object retrieval method (Answer for Question 1):
   Traditional 3D object retrieval methods, both closed-set [1-3] or open-set[4][5] methods, consider only single-layer labels of objects. Besides, traditional open-set methods strictly assume no overlap between the training and testing sets[5][6]. However, in practical real-world scenarios, objects are typically described by multiple hierarchical labels, and the training set and testing set often share a partial space of coarse labels. As shown in Tab. R1 of the rebuttal PDF, we expand the number of label levels in the semi-open learning task, where testing categories are unseen at one level but seen at other levels. The label spaces are disjoint at only one level and have some overlap at other levels.
   [1] Gao Y, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE TIP, 2012.
   [2] He X, et al. Triplet-center loss for multi-view 3d object retrieval[C]. IEEE CVPR, 2018.
   [3] Collins J, et al. Abo: Dataset and benchmarks for real-world 3d object understanding[C]. IEEE CVPR, 2022.
   [4] Zhou Z. Open-environment machine learning[J]. National Science Review, 2022.
   [5] Feng Y, et al. Hypergraph-based multi-modal representation for open-set 3d object retrieval[J]. IEEE TPAMI, 2023.
   [6] Parmar J, et al. Open-world machine learning: applications, challenges, and opportunities[J]. ACM Computing Surveys, 2023.

3. About the efficiency of graphs (Answer for Question 2):
   This paper is an early exploration of semi-open learning. Therefore, we selected the three typical categories for the geometry-based coarse label, which is significantly different from the semantic-based fine category. These three coarse categories and two levels of hierarchical labels are representative of a semi-open environment, helping us focus on exploring the new semi-open learning task and designing a novel collaborative learning paradigm based on hierarchical correlations. Specifically, when the category of coarse labels increases, a natural implementation is to construct a hypergraph structure with more hyperedges to capture more complex correlations. This involves addressing challenges such as complex network representation associated with multiple labels while balancing complexity, efficiency, and performance. Tackling these challenges, we have preliminarily experimented with a hypergraph-based isomorphism computation method for structure compression and complexity reduction, inspired by [7]. Besides, we also have developed a hypergraph-based dynamic system approach to manage the dynamic increase of categories and levels of the label inspired by [8].
   [7] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024.
   [8] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.

4. About the computational requirements comparison (Answer for Question 3):
   Our experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.

5. About the open access to datasets (Answer for Question 4):
   Thanks for your interest in our work. We are well prepared and will release the datasets, code, configurations, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.

Thank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.

Response for Reviewer oLzP

We sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.

1. About the framework (Answer for Weakness 1):
   Based on your suggestions, we have restructured the presentation of the proposed framework. Specifically, the proposed framework consists of two sequentially connected modules: HRE and SET.
   a) The HRE module takes basic features of different modalities as input. This module employs two sets of auto-encoders sequentially to achieve modality fusion and category space retrace, and generates unified embeddings and retrace embeddings for each object.
   a) The SET module takes two types of embeddings from the last module as input, utilizing structure-aware feature smoothing and distillation through hypergraph convolution and memory bank reconstruction, respectively. Finally, this module generates the final features for similar object matching based on feature distance, thereby enabling retrieval.

2. About more comparisons (Answer for Weakness 2):
   Inspired by your suggestions, we added three simpler methods as compared baselines to make our experiments more comprehensive. We provide the additional experimental results in Tab. R3 of the rebuttal PDF. From the results, we can observe that the low performance of simpler methods like MLP and GCN demonstrates the complexity of the semi-open environment and the necessity of research in semi-open learning. The significant improvement achieved by our method also proves its effectiveness. We will include these results and analyses in the revised version of the paper.

3. About citation and writing (Answer for Weakness 3):
   We have added citations for the compared methods as shown in Tab. R3 of the rebuttal PDF, and we have made similar revisions for the tables in the paper.

4. About the level of labels (Answer for Question 1):
   This paper is an early exploration of semi-open learning. Therefore, we selected the two typical levels of labels based on different criteria: geometry-based coarse shape category and semantic-based fine category, which is representative of a semi-open environment. This two-layer framework is also a typical implementation of the collaborative learning paradigm based on hierarchical correlations. When the levels of hierarchical labels increase, a natural implementation is to construct more Retrace Auto-Encoders, and each auto-encoder is designed to retrace one level of the category. This involves addressing challenges such as domain adaptation associated with multiple levels while balancing complexity, efficiency, and performance. Specifically, we have preliminarily experimented with a hypergraph-based isomorphism computation method to address the increase in parameters brought by higher levels, inspired by [1]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of labels at each layer inspired by [2].
   [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2023.
   [2] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.

5. About the computational requirements (Answer for Question 2):
   Our experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.

6. About the data extension (Answer for Question 3):
   As shown in #line 152-153 and Appendix C Algorithm 1, the proposed HERT framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data such as e.g. text, audio, video, 3d, etc. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets, code, configs, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.

Thank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.

Response for Reviewer YDem

We sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.

1. About the technical contribution (Answer for Weakness 1 and the Questions):
   a) A newly general task for real-world practical machine learning. We demonstrated the limitations of naive open-set learning tasks and methods through experiments and proposed a more practical semi-open learning task. Additionally, we constructed four semi-open datasets for benchmarking.
   b) An early explored paradigm and framework for semi-open learning. Specifically, we proposed the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework, including the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules, which are designed to overcome the distribution disequilibrium and confusion of unseen categories in semi-open 3D object retrieval.
   c) A flexible high-order structure for semi-open learning. We propose a superposed Hypergraph structure to capture high-order correlations among objects, under the guidance of local coherent correlations and global entangled correlations from hierarchical category information.
   d) Extensive experiments and analysis. Experimental results on the four datasets demonstrate that our method can outperform state-of-the-art retrieval methods in the semi-open environment.

2. About the task comparison (Answer for Weakness 2):
   We discuss the difference between this new task (semi-open learning) and existing ones (open-set learning) in Tab. R1 of the rebuttal PDF. Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets[1][2]. However, in practical real-world scenarios, objects are typically described by multiple hierarchical labels, and the training set and testing set often share a partial space of coarse labels. We expand the number of label levels in the semi-open learning task, where testing categories are unseen at one level but seen at other levels. The label spaces are disjoint at only one level and have some overlap at other levels.

   [1] Zhou Z. Open-environment machine learning[J]. National Science Review, 2022.
   [2] Parmar J, et al. Open-world machine learning: applications, challenges, and opportunities[J]. ACM Computing Surveys, 2023.

Thank you again for your valuable suggestions, especially your professional advice on presentation and future work in semi-open learning.

Response for Reviewer nF63

We sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.

1. About coarse labels and datasets (Answer for Weakness Major 1, 3, Minor Q1, and the Questions):
   The coarse labels in Figure 1 mean the basic shape of the object as a whole. As shown in Fig. R1 of the rebuttal PDF, we provide examples of four datasets, each with three coarse classes and five fine classes. We annotate the coarse labels according to the geometry-based shape of each object as a whole (#line 242-243, #line 447-449), while ignoring the part assembly relationships within an object in this paper. This paper is an early exploration of semi-open learning. Therefore, we selected two typical levels of labels based on different criteria: geometry-based coarse shape category and semantic-based fine category, which is representative of a semi-open environment. As shown in the right side of Fig. R1, the objects that were removed during dataset construction are those with multiple separated parts and cannot be considered as a whole. We believe that the use of these corner-case samples would intertwine other issues such as graphics and foundation models, and would not reflect the key problem of contradicted hierarchical labels in semi-open learning. Therefore, we decided to exclude them from this early exploration and will address these more complex cases with multiple levels of labels in future work. Additionally, we provide the split of some datasets in Answer 5-7.

2. About multiple levels (Answer for Weakness Major Q2):
   As an early exploration of semi-open learning, we believe this paper should focus on exploring the new semi-open learning task and designing a novel collaborative learning paradigm based on hierarchical correlations. Based on this paradigm, the HERT framework is implemented with two layers temporarily in this paper, aiming to use two of the most representative layers to validate the necessity and performance of machine learning research in typical semi-open environments. However, one of our future work directions is to extend the HERT framework to encompass more intertwined factors in complex semi-open environments. This involves addressing challenges such as domain adaptation associated with multiple levels while balancing complexity, efficiency, and performance. Specifically, we have preliminarily experimented with a hypergraph-based isomorphism computation method to address the increase in parameters brought by higher levels, inspired by [1]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of labels at each layer inspired by [2].
   [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2023.
   [2] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.

3. About failure cases (Answer for Weakness Minor Q3):
   We provide the visualization of failure cases in Fig. R3 of the rebuttal PDF. In these failure cases, the query objects (bench, TV stand, bookshelf) and the wrong-matched target objects (mantel, laptop) share a certain similarity in their shapes and belong to the same coarse category (Rectangular-Cubic Prism). Although the significant performance improvement of the HERT framework demonstrates the necessity of research in semi-open learning and the effectiveness of our method, these corner cases also indicate the necessity of utilizing finer-level information such as the part-assembly of objects. This issue is the same as the multiple levels issue mentioned in Weakness Major Q2. However, this paper focuses more on the fundamental differences brought by semi-open hierarchical labels. Therefore, we only consider two layers of labels in this study. As mentioned in the Answer 2 above, we are currently conducting research to address these more complex environments. Thank you for your keen observations and academic insights.

4. About the writing (For Weakness Minor Q2):
   Thanks for your thorough review and suggestions. We will conduct a comprehensive review of the entire paper, especially focusing on the use of abbreviations.

5. Splits of SO-NTU:
   5.1 Train
   Rectangular-Cubic Prism: headstone,table square
   Solids of Revolution:ball,ballon,cannon,watch
   Miscellaneous: book,plant with pot,cold weapon stick,frame,gun pistol,plane delta wing,plant leaf
   5.2 Test
   Rectangular-Cubic Prism: Bed,truck,chair common,sofa,man,chair,computer,container,tank
   Solids of Revolution: fish,bottle,knife,cup,helmet,hydrant,insect fly,insect polypod,missle,orchestral,pen,plane backswept wing,plane forwardswept wing,tree,weed,ring,table round,hammer,screwdriver,wheel,zeppelin
   Miscellaneous: dinosaur,dog,duck,tetrapods,bird,car common,chair swivel,chess,chip,clock,cold weapon long,sword,cycle bike,cycle moto,door,gun musket,gun submachine,human stand,floorlamp,table lamp,giant,helicopter,straight wing,flower,galleon,ship modern

6. Splits of SO-MN40:
   6.1 Train
   Rectangular-Cubic Prism: table,night stand,sink, monitor
   Solids of Revolution: Glass box,flower pot
   Miscellaneous: Keyboard, airplane
   6.2 Test
   Rectangular-Cubic Prism: Mantel,tv stand,desk,sofa,bed,bookshelf,chair,bathtub,wardrobe,dresser, radio,piano,bench,xbox,range hood
   Solids of Revolution: Bottle,bowl,cup,stool,vase,cone,tent
   Miscellaneous: Toilet,curtain,car,guitar,stairs,door,person,laptop,plant,lamp

7. Splits of SO-ABO:
   7.1 Train
   Rectangular-Cubic Prism: table Solids of Revolution: tent Miscellaneous: Mirror,Plant or flower pot
   7.2 Test
   Rectangular-Cubic Prism: chair,cart,shelf,cabinet,dresser,bed,bench,ladder,sofa Solids of Revolution: exercise weight,container or basket,vase,ottoman,pillow Miscellaneous: picture frame or painting,lamp,fan

Thank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.

Dear Reviewer,

We would greatly appreciate any updates or feedback you might have regarding our responses to your initial comments. Your insights are valuable to us as we work to improve our paper.

If you need any additional information or clarification from our side, please don't hesitate to let us know.

Thank you for your time and consideration.

Thanks again for your valuable suggestions. We will respond to your comments separately in the following two text boxes due to space limitations.

1. About ottoman in SO-ABO and plants with pots in SO-NTU

• For the ottoman: Since the shapes of the ottomans are typically composed of curved surfaces and are closer to ellipsoids, we removed the few cube-shaped samples from the ottoman category in the ABO dataset when constructing the SO-ABO dataset, and kept only the ellipsoid-like samples for the experiment.

• For the plants with pots: The samples in this category are not pots but rather a variety of plants with diverse and unusual shapes (such as Lavender, Snake Plant, Jasmine, Spider Plant, Aloe Vera, etc.), along with their pots, which also have different shapes. Therefore, we classified these objects under Miscellaneous in this paper.

  We will provide more examples and explainations, especially for these categories, in the revised version.

2. About the Objaverse and Miscellaneous Category

   Thank you for the reminder, and we apologize for omitting the necessary emphasis and analysis for Objaverse in our first round of response. Objaverse is an outstanding work, and we believe that it is one of the most important and practically datasets in the 3D vision field in recent decades. This dataset provides a richer quantity of objects, finer categories, and diverse domains, with a greater diversity of 3D shapes. It is an essential resource for further exploring more complex and practical semi-open learning. We will provide a detailed analysis comparing our datasets with Objaverse, a discussion of Objaverse's irreplaceable role in semi-open learning, and necessary references [1] in the revised version of this paper.

    [1] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects[C]. Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.  
   

   However, our work represents an early exploration of the semi-open environment. To explore this objectively existing environment, we have made a preliminary construction of the four datasets (SO-ESB, SO-NTU, SO-MN40, and SO-ABO). Within the context of these datasets, we used coarse labels such as Rectangular-Cubic Prism and Solids of Revolution to describe two categories of objects with typical coarse shapes. For other objects in the datasets that lack the typical shapes features, we temporarily classified them under the Miscellaneous category. The experimental results demonstrate that even with these simple coarse labels, the semi-open environment exists objectively and our method also achieved significant improvements. We believe that it is necessary in this paper to focus on typically modeling this new environment setting, analyzing the essential key challenges while filtering out the effects of other atypical or random noise. We acknowledge that conducting a comprehensive study of an entirely new field may be difficult within the scope of a conference paper.

   Inspired by your comment, we will focus our future work on more hierarchical levels of labels in semi-open learning. We plan to dedicate more effort to exploring Objaverse and other complex datasets, investigating various shape, domain, and semantic labels beyond the shape-based coarse labels proposed in this paper. This will allow us to better adapt to more complex objects and advance the practical applications of semi-open learning. We are well prepared and will release the datasets, code, configurations, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Looking forward to academic discussions with you after the anonymous period of NeurIPS 24, if possible! We are willing to share all our experiences, dataset, and code of this work.

3. About the Framework
   As mentioned in the answer above, the two-layer HERT is designed based on the existing typical semi-open assumption, serving as an early exploration setting of semi-open learning. The hypergraph-based isomorphism computation and dynamic system are the extended modules of the initial HERT framework for practical applications, we will provide more detailed experimental results and analysis in the revised version of this paper. Here, we provide some results and analysis:

• For the isomorphism computation
  In order to handle more levels of labels, we simplified the hypergraph construction process using isomorphism computation following [2]. Specifically, we detect and merge hyperedges with similar structures or edge embeddings within the proposed HERT framework, thereby reducing the complexity of the hypergraph. To evaluate the efficiency of this approach, we conduct the compared experiments between frameworks with and without isomorphism computation on SO-MN40 datasets. As shown in the table below, retrieval accuracy (mAP) improved as the number of layers increased, but both training and inference times also increased. Isomorphism computation significantly improved the efficiency of training and inference, with more notable gains in efficiency and accuracy observed as the layers increased.

  Table R4: Ablation studies of isomorphism computation on SO-MN40 dataset

  number of layers	2	3	4
  mAP w/o IC	0.6336	0.6441	0.6583
  mAP with IC	0.6359	0.6483	0.6674
  mAP Improvement	0.36%	0.65%	1.38%
  Training Time (s) w/o IC	91.16	95.79	99.73
  Training Time (s) with IC	89.31	92.49	94.97
  Training Efficiency Improvement	2.51%	3.44%	4.77%
  Inference Time (ms) w/o IC	18.42	19.51	20.76
  Inference Time (ms) with IC	17.37	17.74	18.31
  Inference Efficiency Improvement	5.70%	9.07%	11.80%

  w/o denote without, IC denotes the isomorphism computation module

• For the dynamic system
  To handle the increasing number of labels, we employed a hypergraph-based dynamic system to incrementally construct the hypergraph following [3]. Specifically, we constructed hyperedges for new vertices of new samples and new labels and updated the hypergraph structure within the proposed HERT framework. To evaluate the efficiency of this approach, we conduct the compared experiments between frameworks with and without the dynamic system on SO-MN40 datasets. As shown in the table below, retrieval accuracy (mAP) improved as the number of coarse categories increased, while an increase in the number of fine categories slightly decreased retrieval accuracy. Both increases in coarse and fine categories led to longer training and inference times. The dynamic system significantly improved the efficiency of training and inference, with more notable gains in efficiency and accuracy observed as the categories increased.

  Table R5: Ablation studies of dynamic system on SO-MN40 dataset

  Number of Categories	Original HERT	Coarse: 3->4	Coarse: 3->5	Fine: 32->40	Fine: 32->49
  mAP w/o DS	0.6336	0.6395	0.6427	0.6295	0.6253
  mAP with DS	-	0.6431	0.6478	0.6331	0.6327
  mAP Improvement	-	0.56%	0.79%	0.57%	1.18%
  Training Time(s) w/o DS	91.60	93.83	95.31	93.98	95.57
  Training Time(s) with DS	-	91.45	91.61	91.37	91.59
  Training Efficiency Improvement	-	2.53%	3.88%	2.78%	4.17%
  Inference Time(ms) w/o DS	18.42	19.74	20.01	19.93	21.36
  Inference Time(ms) with DS	-	18.45	18.39	18.57	18.43
  Inference Efficiency Improvement	-	6.50%	8.10%	6.82%	13.72%

  w/o denotes without, DS denotes the dynamic system module, and Original HERT means the original framework designed for 3 coarse categories and 32 fine categories. '->' denotes the increase in category number.

• Conclusion
  Experimental results above demonstrate that isomorphism computation and the dynamic system can effectively enhance the efficiency of the HERT framework, and they have the potential to advance the practical application of semi-open learning methods. we will provide more detailed results and analysis in the revised version of this paper.

Thank you again for your valuable suggestions, especially your professional advice on practical applications in semi-open learning.