LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100x Faster Inference

Response to W5

We apologize for the unclear descriptions. Compared to the graph KD methods, our LightHGNN can capture high-order information. This is achieved by the proposed high-order reliable quantification, high-order correlation sampling, and high-order soft-target constraint. First, our high-order reliable quantification module can endow each hyperedge with a reliable score by evaluating its capability to combat noise. Second, our high-order correlation sampling module can sample the high-order correlations from the hypergraph via the reliable score, which is implemented by sampling from the Bernoulli distribution. Third, our high-order soft-target constraint can supervise the learned embeddings from the student MLPs and explicitly pull them close to the soft labels of vertices from those reliable hyperedges. All three modules can help the student MLPs capture the high-order information, thus yielding better performance on hypergraph datasets. We will add those descriptions in the revised version.

We also summarize the challenges of hypergraph KD compared to graph KD, which states the necessity of exploring the hypergraph KD methods.

• The beyond pairwise modeling capability brings more challenges. The hypergraph is a generalization of the graph, which can model the low-order structure (graph) and high-order structure. Thus, compared to the distilling graph neural networks that only consider the pair-wise structure, distilling hypergraph neural networks will consider both low-order and high-order structures, which brings more challenges to the distillation of hypergraph neural networks.
• The multi-model property of hypergraph brings more challenges. Hypergraph can naturally model the multi-modal correlation via the incidence matrix concatenation [1], which is more flexible than the graph. In real-world applications, the complex high-order correlations are inherently multi-modal. For example, in the social network, the people and relationships are represented by vertex and hyperedge, and the people can be divided into different groups, and the relationships can be divided into different types, like family, friends, and colleagues. Clearly, distilling those complex (multi-modal) high-order correlations (hypergraph neural networks) is more challenging than the pairwise correlation (graph neural networks).
• The information of the hypergraph is hard to quantify. As for distilling graph neural networks, the information of the graph is easy to quantify. For example, the degree of the edge is fixed to 2, and the connection of the vertex is simple (single modality and directly propagating messages from other vertices). In contrast, the information in the hypergraph is hard to quantify. For example, the degree of the hyperedge is not fixed (one, two, or more are all legal), and the connection of the vertex is complex (multi-modal and propagating messages from other vertices or hyperedge or group vertices). Thus, controlling the process of distilling hypergraph neural networks or making a reliable hypergraph neural network distillation is more challenging than distilling graph neural networks.

As demonstrated in [2], GCN [3] is a special case of HGNN confronting the pair-wise correlation. HGNN is more powerful than GNN but also suffers from the higher runtime and memory cost. Thus, to better capture those high-order correlations in the hypergraph, we propose a reliable hyperedge quantification and sampling strategy to achieve topology-aware distillation for hypergraph neural networks. This work is the first attempt at distilling hypergraph neural networks. In the future, we will consider more downstream tasks, more modalities, and more strategies of reliable quantification for distilling hypergraph neural networks.

[1] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

[2] Gao et al. HGNN$^+$: General Hypergraph Neural Networks. TPAMI, 2023.

[3] Kipf et al. Semi-supervised classification with graph convolutional networks. ICLR, 2017.

Response to W4

Thanks for your suggestions. We further conduct experiments to demonstrate the capability of our LightHGNNs on large-scale hypergraph inference. Two large scale hypergraph datasets[1]: Recipe-100k (10w vertices) and Recipe-200k (24w vertices) are employed here. The two datasets both contain eight categories. Specifically, the Recipe-100k dataset contains 101,585 vertices and 12,387 hyperedges, and the Recipe-200k dataset contains 240,094 vertices and 18,129 hyperedges. The vertex features are the bag of words from the sentence that makes the recipe. Hyperedges are the ingredients of the recipe or the Keywords for food preparation steps. Experiments on the two datasets are shown in the following table.

As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more obvious. Naive HGNN[2] becomes slower and slower, even throwing the Out-Of-Memory error confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework. The results indicate that our LightHGNNs are more suitable for large-scale hypergraph inference.

[1] Li et al. SHARE: a System for Hierarchical Assistive Recipe Editing. EMNLP, 2022.

[2] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

Response to W1

Thanks for your suggestions. We further add the experiments of NOSMOG[1] in the table of "Table 3: Experimental results on graph and hypergraph datasets.". A part of the results are shown in the following table.

As shown in the table, our LightHGNNs achieve comparable performance to NOSMOG[1] on graph datasets and significantly better performance on hypergraph datasets. The results demonstrate the effectiveness of our distillation framework confronting both graph and hypergraph datasets.

[1] Tian et al. Learning mlps on graphs: A unified view of effectiveness, robustness, and efficiency. ICLR, 2023.

Response to W2

Thanks for your comments. We attribute the performance improvement of LightHGNNs to the following two reasons. First, knowledge distillation from teacher HGNN determines the student LightHGNNs can achieve comparable performance to HGNN. Second, the proposed hyperedge reliability sampling can help the student MLPs distinguish those noise hyperedges in the original hypergraph, which can help the student LightHGNN$^+$ capture more task-related and reliable information, alleviate over-smoothing (as empirically demonstrated in Section 5.4), thus yielding better performance.

Response to W3

Thanks for your suggestions. We will future clarify the difference between our work and KRD[1] and add proper citations in the revised version. The main difference between our work and KRD[1] is two-fold.

• KRD [1] is a typical reliable-based distillation for graph neural networks. However, its student MLPs can only be supervised by the soft label from those reliable vertices (knowledge points in their paper), which still lose the structure information and cannot be utilized in the hypergraph. However, our LightHGNN$^+$ can quantify the reliability of the high-order correlations and inject the reliable high-order information into the student MLPs via explicit supervision from high-order soft labels of those reliable hyperedges. The proposed LightHGNN$^+$ can sufficiently take advantage of both the vertex features and high-order structure information, thus yielding better performance.
• Besides, we design a metric to measure the over-smoothing phenomenon (the relevance of features and hypergraph typology) in hypergraph neural networks and empirically demonstrate that our LightHGNN$^+$ has the capability to combat the over-smoothing and achieve better performance, which is not considered in KRD[1].

All those discussions will be added in the revised version.

[1] Zhang et al. Graph-less neural networks: Teaching old MLPs new tricks via distillation. ICLR, 2022.

Response to Q3

Thanks for your comments. Actually, in our experiment section 5.4, we attempt to define a specific metric to quantify the over-smoothing phenomenon. Here, we design a topology-aware score $\mathcal{S}$ to measure the relevance of features and hypergraph typology. Specifically, we first calculate the cosine similarity between the vertex features and the hyperedge features, and then calculate the average cosine similarity of all vertices in the hypergraph. The topology-aware score $\mathcal{S}$ is defined as the average cosine similarity of all hyperedges in the hypergraph. The higher the score, the more relevant the features and hypergraph typology. We first provide the accuracy of MLP, HGNN and LightHGNNs with respect to different number of layers, as shown in the following table.

As shown in the table, the accuracy of MLP, HGNN, and LightHGNNs all significantly decrease as the number of layers increases, especially for the MLP and HGNN. However, the accuracy of LightHGNN+ is more stable than MLP, HGNN, and LightHGNNs. This is because the proposed LightHGNN+ can alleviate the over-smoothing phenomenon via the hyperedge reliability sampling, thus yielding robust performance combating the increase of the number of layers. We further provide the topology-aware score $\mathcal{S}$ of MLP, HGNN and LightHGNNs with respect to different number of layers, as shown in the following table.

As shown in the above table, we have the following three observations. First, despite the increase in the number of layers, the topology-aware score of MLP is always higher than that of HGNN and LightHGNNs. This is because the MLP only relies on the vertex features and ignores the hypergraph typology. Second, the topology-aware score of LightHGNN+ is more stable than MLP, HGNN, and LightHGNNs. This is because the proposed LightHGNN+ can alleviate the over-smoothing phenomenon via the hyperedge reliability sampling, thus yielding robust performance combating the increase of the number of layers. Third, we find an interesting phenomenon that the topology-aware score significantly decreases in #layer 6->7. The results explain when the accuracy of HGNN and LightHGNN significantly decreases in #layer 6->7, as shown in the above table. This is because the phenomenon of over-smoothing has reached a critical point with the increase in the number of layers, impacting the representations for hypergraphs. As a result, the performance has sharply declined. However, why the critical point is reached at this specific layer needs further investigation in future work. I believe this is an intriguing experimental result that will impact the research community's study of over-smoothing.

Response to Q4

Thanks for your insightful comments. It's important to note that the over-smoothing metric we introduced is a preliminary attempt to capture the phenomenon. The defined metrics attempt to measure the relevance of features and hypergraph typology. However, the over-smoothing phenomenon is a complex problem, which can be caused by various reasons like the dataset, model architecture, and optimization strategy. Our typology-aware scoring is a preliminary attempt, and there is much work to be done further, such as considering the weights between different nodes within the same hyperedge and considering the influence of the high-order neighbor smoothing. These avenues provide various opportunities for researchers to delve deeper into the intricacies of over-smoothing and its implications for representation learning on hypergraphs.

Response to Q5

Thanks for your insightful comments. In the field of knowledge distillation for Hypergraph Neural Networks (HGNNs), the findings regarding resistance to over-smoothing could impact future research directions and practical applications in the following aspects:

1. Model Robustness: Research on resistance mechanisms to over-smoothing can aid in developing more robust HGNN models that maintain feature diversity across multiple network layers and avoid homogenization of features in the deeper layers.

2. Deep Network Design: Understanding how to control and mitigate over-smoothing is crucial for designing deeper HGNN structures while preserving high-order feature information, enhancing the expressiveness of the models.

3. Computational Efficiency: Over-smoothing resistance techniques could reduce the number of iterations needed during distillation, as they help avoid rapid convergence to suboptimal states. This directly impacts the computational efficiency and inference speed of models, especially on large-scale datasets.

4. Optimization Strategy Research: Research into over-smoothing could inspire new optimization strategies, such as regularization techniques or novel loss functions, which could reduce the issue of over-smoothing while maintaining model performance.

These implications suggest that research into over-smoothing resistance is not only vital for understanding HGNNs but also for enhancing the practical deployment value of the distilled student models.

Response to Q6

Thanks for your suggestions. We further conduct experiments to demonstrate the capability of our LightHGNNs on large-scale hypergraph inference. Two large scale hypergraph datasets[1]: Recipe-100k (10w vertices) and Recipe-200k (24w vertices) are employed here. The two datasets both contain eight categories. Specifically, the Recipe-100k dataset contains 101,585 vertices and 12,387 hyperedges, and the Recipe-200k dataset contains 240,094 vertices and 18,129 hyperedges. The vertex features are the bag of words from the sentence that makes the recipe. Hyperedges are the ingredients of the recipe or the Keywords for food preparation steps. Experiments on the two datasets are shown in the following table.

As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more obvious. Naive HGNN[2] becomes slower and slower, even throwing the Out-Of-Memory error confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework. The results indicate that our LightHGNNs are more suitable for large-scale hypergraph inference.

[1] Li et al. SHARE: a System for Hierarchical Assistive Recipe Editing. EMNLP, 2022. [2] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

Response to W3

Thanks for your suggestions. We further conduct experiments on two large-scale hypergraph datasets[1]: Recipe-100k (10w vertices) and Recipe-200k (24w vertices). The two datasets both contain eight categories. Specifically, the Recipe-100k dataset contains 101,585 vertices and 12,387 hyperedges, and the Recipe-200k dataset contains 240,094 vertices and 18,129 hyperedges. The vertex features are the bag of words from the sentence that makes the recipe. Hyperedges are the ingredients of the recipe or the Keywords for food preparation steps. Experiments on the two datasets are shown in the following table.

As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more obvious. Naive HGNN[2] becomes slower and slower even throwing Out-Of-Memory error confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework. The results indicate that our LightHGNNs are more suitable for large-scale hypergraph inference.

[1] Li et al. SHARE: a System for Hierarchical Assistive Recipe Editing. EMNLP, 2022.

[2] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

Response to Q1

Thanks for your comments. The main reason for choosing the baseline HGNN is that it is a typical and widely used neural network on hypergraphs. It is also a direct extension of the typical graph neural networks [1] as demonstrated in Gao et al. [2]. Since the proposed LightHGNNs is a general distillation framework for hypergraph neural networks, we also provide experiments on other advanced hypergraph neural networks, including HGNN+[2], UniGNN, UniGAT, UniGCN2[3], ED-HNN[4], and AllSet[5], as shown in the table of "Response to W1".

[1] Kipf et al. Semi-supervised classification with graph convolutional networks. ICLR, 2017.

[2] Gao et al. HGNN$^+$: General Hypergraph Neural Networks. TPAMI, 2023.

[3] Huang et al. Unignn: a unified framework for graph and hypergraph neural networks. IJCAI, 2021.

[4] Wang et al. Equivariant Hypergraph Diffusion Neural Operators, ICLR, 2023.

[5] Chien et al. You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, ICLR, 2022.

Response to Q2

Thanks for your comments. In this paper, we propose a hypergraph neural networks distillation framework, which does not rely on the specific hypergraph neural networks selection. The proposed LightHGNN can be applied to any hypergraph neural network to achieve fast and accurate inference, as shown in the table of "Response to W1".

Response to W1

Thanks for your comments. In the submission, we have provided the experiments on different teacher architectures, including HGNN, HGNN+, UniGNN, UniGAT, in Appendix H and Table S9. Here, following the reviewer's suggestions, we further conduct experiments with more advanced methods including UniGCN2[1], ED-HNN[2], and AllSet[3]. Experimental results are shown in the following table.

As shown in the table, as the teacher model becomes more advanced, the performance of our LightHGNNs also becomes better. We will add those results in Table S9 "Experimental results of LightHGNNs with different teacher architectures." in Appendix H.

[1] Huang et al. Unignn: a unified framework for graph and hypergraph neural networks. IJCAI, 2021.

[2] Wang et al. Equivariant Hypergraph Diffusion Neural Operators, ICLR, 2023.

[3] Chien et al. You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, ICLR, 2022.

Response to W2

Thanks for your suggestions. We further conduct experiments on the number of hidden layers of MLP, HGNN and LightHGNNs. The results are shown in the following table.

As shown in the table, the accuracy of MLP, HGNN, and LightHGNNs all significantly decrease as the number of layers increases, especially for the MLP and HGNN. However, the accuracy of LightHGNN+ is more stable than MLP, HGNN, and LightHGNNs. This is because the proposed LightHGNN+ can alleviate the over-smoothing phenomenon via the hyperedge reliability sampling, thus yielding robust performance combating the increase of the number of layers.

Response to Q1

Thanks for your comments. We attribute the performance improvement of LightHGNN$^+$ to the following two reasons. First, knowledge distillation from teacher HGNN determines the student LightHGNNs can achieve comparable performance to HGNN. Second, the proposed hyperedge reliability sampling can help the student MLPs distinguish those noise hyperedges in the original hypergraph, which can help the student LightHGNN$^+$ capture more task-related and reliable information, alleviate over-smoothing (as empirically demonstrated in Section 5.4), thus yielding better performance.

Response to Q2

Thanks for your comments. Actually, in most datasets under the transductive setting, LightHGNN$^+$ performs better than LightHGNN. There are still three datasets (News20, CA-Cora, and DBLP-Conf), where LightHGNN$^+$ performs worse than LightHGNN and HGNN. This is because, as shown in the following table, the number of hyperedges in the three datasets is small, which limits the power of our reliability hyperedge quantification and sampling module. However, as the number of hyperedges increases, the performance of LightHGNN$^+$ becomes better and better. As shown in the above table, our LightHGNN$^+$ can achieve better performance than LightHGNN and HGNN in the Recipe-100k and Recipe-200k datasets. Thus, our LightHGNN$^+$ has more potential to be applied to large-scale hypergraph inference and achieve better performance.

Response to Weaknesses

Thanks for your comments. We apologize for the unclear description of the novelty of this paper. In the following, we clarify the novelty of the proposed LightHGNNs from two perspectives.

• Compared to GNNs-to-MLPs Methods. First, the Graph-Less Neural Networks (GLNN) [1] proposes using the soft label of the teacher GNNs to supervise the student MLPs. Compared to it, our LightHGNN is a simple extension from GNNs to HGNNs. However, our LightHGNN$^+$ further proposes the high-order soft label to help the student MLPs learn more high-order structure information from the teacher HGNNs. Second, as for the Knowledge-inspired Reliable Distillation (KRD) [2], its student MLPs can only be supervised by the soft label from those reliable vertices (knowledge point in their paper), which still lose the structure information and cannot be utilized in the hypergraph. However, our LightHGNN$^+$ can quantify the reliability of the high-order correlations and inject the reliable high-order information into the student MLPs via explicit supervision from high-order soft labels of those reliable hyperedges. The proposed LightHGNN$^+$ can sufficiently take advantage of both the vertex features and high-order structure information, thus yielding better performance and faster inference speed.
• Compared to HGNNs Methods. To the best of my belief, our work is the first to distill hypergraph neural networks to MLPs. Our LightHGNNs can achieve comparable and even better performance in hypergraph datasets with much faster inference speed and lower memory cost. Those capabilities allow the LightHGNNs to be applied to large-scale hypergraph inference, which can not be achieved by the naive HGNNs. In the following table, we conduct experiments on two larger hypergraph datasets[3]: Recipe-100k (10w vertices) and Recipe-200k (24w vertices).

As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more obvious. Naive HGNN[4] becomes slower and slower even throwing Out-Of-Memory error confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework.

[1] Zhang et al. Graph-less neural networks: Teaching old MLPs new tricks via distillation. ICLR, 2022.

[2] Wu et al. Quantifying the knowledge in gnns for reliable distillation into MLPs. ICML, 2023.

[3] Li et al. SHARE: a System for Hierarchical Assistive Recipe Editing. EMNLP, 2022.

[4] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

Response to Q5

Thanks for your comments. Compared with the distilling graph neural networks (GNN), the challenges of distilling hypergraph neural networks (HGNN) are as follows.

• The beyond pairwise modeling capability brings more challenges. The hypergraph is a generalization of the graph, which can model both the low-order structure (graph) and high-order structure. Thus, compared to the distilling graph neural networks that only consider the pair-wise structure, distilling hypergraph neural networks will consider both low-order and high-order structures, which brings more challenges to the distillation of hypergraph neural networks.
• The multi-model property of hypergraph brings more challenges. Hypergraphs can naturally model the multi-modal correlation via the incidence matrix concatenation [1], which is more flexible than the graph. In real-world applications, the complex high-order correlations are inherently multi-modal. For example, in the social network, the people and relationships are represented by vertex and hyperedge, and the people can be divided into different groups, and the relationships can be divided into different types, like family, friends, and colleagues. Clearly, distilling those complex (multi-modal) high-order correlations (hypergraph neural networks) is more challenging than the pairwise correlation (graph neural networks).
• The information of the hypergraph is hard to quantify. As for distilling graph neural networks, the information of the graph is easy to quantify. For example, the degree of the edge is fixed to 2, and the connection of the vertex is simple (single modality and directly propagating messages from other vertices). In contrast, the information in the hypergraph is hard to quantify. For example, the degree of the hyperedge is not fixed (one, two, or more are all legal), and the connection of the vertex is complex (multi-modal and propagating messages from other vertices or hyperedge or group vertices). Thus, controlling the process of distilling hypergraph neural networks or making a reliable hypergraph neural network distillation is more challenging than distilling graph neural networks.

As demonstrated in [2], GCN [3] is a special case of HGNN confronting the pair-wise correlation. HGNN is more powerful than GNN but also suffers from the higher runtime and memory cost. Thus, to better capture those high-order correlations in the hypergraph, we propose a reliable hyperedge quantification and sampling strategy to achieve topology-aware distillation for hypergraph neural networks. This work is the first attempt at distilling hypergraph neural networks. In the future, we will consider more downstream tasks, more modalities, and more strategies of reliable quantification for distilling hypergraph neural networks.

[1] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

[2] Gao et al. HGNN$^+$: General Hypergraph Neural Networks. TPAMI, 2023.

[3] Kipf et al. Semi-supervised classification with graph convolutional networks. ICLR, 2017.

Response to Q6

Thanks for your comments. Here, we discuss our significance compared to (Wu et al. 2023)[1]. We will further clarify the difference between our work and KRD[1] and add proper citations in the revised version. The main difference between our work and KRD[1] is two-fold.

• KRD [1] is a typical reliable-based distillation for graph neural networks. However, its student MLPs can only be supervised by the soft label from those reliable vertices (knowledge points in their paper), which still lose the structure information and cannot be utilized in the hypergraph. However, our LightHGNN$^+$ can quantify the reliability of the high-order correlations and inject the reliable high-order information into the student MLPs via explicit supervision from high-order soft labels of those reliable hyperedges. The proposed LightHGNN$^+$ can sufficiently take advantage of both the vertex features and high-order structure information, thus yielding better performance.
• Besides, we design a metric to measure the over-smoothing phenomenon (the relevance of features and hypergraph typology) in hypergraph neural networks and empirically demonstrate that our LightHGNN$^+$ has the capability to combat the over-smoothing and achieve better performance, which is not considered in KRD[1].

All those discussions will be added in the revised version.

[1] Wu et al. Quantifying the knowledge in gnns for reliable distillation into MLPs. ICML, 2023.

Response to W3 and Q3

Thanks for your comments. As for the question of "generalizability of the proposed method across a wider range of scenarios", we know that the distillation framework is highly dependent on the task. Different tasks need different distillation strategies. For example, the distillation of image classification[1] is to pull the KL Divergence of the teacher's and student's prediction of the classification, which is not suitable for the image retrieval task[2]. We follow the typical distillation frameworks on graphs [3][4], focusing on the vertex classification task. We thank the reviewer's suggestions. It is crucial to seek a general distillation framework for a wide range of different tasks, like vertex classification, hyperedge classification, and hyperedge prediction. We will consider it in our future work.

As for the question of "more types of hypergraph datasets", we will add two larger scale hypergraph datasets in the final version, which model the high-order correlation among foods. The two datasets [5] both contain eight categories. Specifically, the Recipe-100k dataset contains 101,585 vertices and 12,387 hyperedges, and the Recipe-200k dataset contains 240,094 vertices and 18,129 hyperedges. The vertex features are the bag of words from the sentence that makes the recipe. Hyperedges are the ingredients of the recipe or the Keywords for food preparation steps. Experiments on the two datasets are shown in the following table.

As shown in the table, our LightHGNNs can not only achieve better performance but also extremely reduce the inference time. As the scale of the dataset increases, the advantage of our LightHGNNs becomes more obvious. Naive HGNN[6] becomes slower and slower, even throwing Out-Of-Memory errors confronting 10w+ vertices, while our LightHGNNs are still fast. Besides, our LightHGNNs exhibit better performance compared to the naive HGNN under the transductive testing set (about 1w or 2w vertices for testing), which demonstrates the effectiveness of our distillation framework. The results indicate that our LightHGNNs are more suitable for large-scale hypergraph inference.

[1] Hinton et al. Distilling the knowledge in a neural network. NIPS, 2014.

[2] Chen et al. Simplified TinyBERT: Knowledge Distillation for Document Retrieval. ECIR, 2021

[3] Zhang et al. Graph-less neural networks: Teaching old MLPs new tricks via distillation. ICLR, 2022.

[4] Wu et al. Quantifying the knowledge in gnns for reliable distillation into MLPs. ICML, 2023.

[5] Li et al. SHARE: a System for Hierarchical Assistive Recipe Editing. EMNLP, 2022.

[6] Feng et al. Hypergraph Neural Networks. AAAI, 2019.

Response to Q4

Thanks for your comments. We apologize for the unclear description of how to apply the GNN-distillation methods to hypergraph-structured datasets in the experiments. Since GNN-distillation methods cannot be directly applied to the hypergraph, we first transform the hypergraph into a graph by the typical clique expansion, following [1]. Then, we apply the GNN-distillation methods to the generated graph for training and testing. We have provided the details in Appendix F.

[1] Gao et al. HGNN$^+$: General Hypergraph Neural Networks. TPAMI, 2023.

Response to W2 and Q2

Thanks for your comments. In this work, we propose a fast-inference framework (including LightHGNN and LightHGNN+) for hypergraph neural networks, whose teachers can be arbitrary hypergraph neural networks. In the following, we provide more experiments of "advanced ones" to show the effectiveness of our distillation framework. As shown in the following table, as the teacher model becomes more advanced, the performance of our LightHGNNs also becomes better.

In the table, UniGCN2 [1], ED-HNN [2], and AllSet [3] are the added "advanced ones" for comparison. We will add those results in Table S9 "Experimental results of LightHGNNs with different teacher architectures." in the Appendix.

[1] Huang et al. Unignn: a unified framework for graph and hypergraph neural networks. IJCAI, 2021.

[2] Wang et al. Equivariant Hypergraph Diffusion Neural Operators, ICLR, 2023.

[3] Chien et al. You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks, ICLR, 2022.

Response to W1 and Q1

Thanks for your comments. In the following, we use (a) and (b) to denote the question "the importance of reliability-based sampling" and the question "the effectiveness of the proposed methodology of measuring the reliability", respectively.

From theoretical point of view. The sampling process of (a) is a typical Bernoulli distribution, whose effectiveness relies on the estimation of hyperedge's reliability (b). In our work, each hyperedge's reliability is generated by the aggregation of linked vertices' variance of entropy. Clearly, the variance of entropy is a good indicator of the reliability of a vertex, which reflects the capability of resisting to noise, as proved by Wu et al.(2023). Thus, in this paper, we extend the concept of vertex's reliability to hyperedge's reliability for high-order correlation reliabile quantification.

From experimental point of view. Follow the review's suggestion, we further conduct ablation study on "utilizing all hyperedges (a)" and "relying on vertex reliability (b)", as shown in the following table.

Here, we adopt three hypergraph datasets for experimental validation. The last two columns denote the point (a) and (b), respectively. We have the following two observations. First, we find that the LightHGNN$^+$ (hyperedge reliability) and (b) (vertex reliability) both perform better than LightHGNN without any reliability quantification, which indicates that the reliability quantification of vertex and hyperedge is useful for distilling knowledge from teacher to student. Second, the results of (a) perform worse than both LightHGNN and LightHGNN$^+$. This is because utilizing all hyperedges for high-order knowledge injection will lead to noise injection and over-smoothing problems. Since those hyperedges already learned by the teacher HGNN, we here only need to inject the knowledge of those reliable and task-related hyperedges into the student MLPs. Thus, the proposed hyperedge reliability quantification is crucial for the effectiveness of our distillation framework. This is also the reason that our LightHGNN$^+$ can achieve better performance than LightHGNN.