DistillHGNN: A Knowledge Distillation Approach for High-Speed Hypergraph Neural Networks

We sincerely appreciate your detailed and thoughtful feedback, which has significantly contributed to enhancing the quality of our manuscript.

Weaknesses:

1. We agree that the caption could be more concise while retaining clarity. In the revised version, we have shortened the caption of Figure 1 to highlight the essential elements.

2. We have revised all direct author references throughout the paper to use citet{} instead of cite{}.

3. We have expanded Table 3 to include key structural properties of the six datasets, providing context for the experiments:

D1) CC-Citeseer: Sparse, homogeneous citation network (3,312 nodes, 1,004 hyperedges, avg. degree 3.2), testing robustness to sparse, biased connections.

D2) CC-Cora: Moderately sparse, homogeneous network (2,708 nodes, 1,483 hyperedges, avg. degree 3.8), highlighting modular structures.

D3) IMDB-AW: Dense, heterogeneous actor-movie network (5,355 nodes, 6,811 hyperedges, avg. degree 8.4), suitable for testing inductive learning in dynamic scenarios.

D4) DBLP-paper: Heterogeneous paper-author network (14,376 nodes, 14,475 hyperedges, avg. degree 5.2), focusing on collaborative dynamics.

D5) DBLP-term: Dense, heterogeneous term-paper network (14,376 nodes, 13,789 hyperedges, avg. degree 7.1), capturing semantic clustering.

D6) DBLP-Conf: Sparse, heterogeneous conference-paper network (14,376 nodes, 1,612 hyperedges, avg. hyperedge size 284.2), emphasizing hierarchical organisation.

D7) DBLP: Large, dense, heterogeneous network (66,543 nodes, 274,824 hyperedges, avg. degree 8.26), exploring collaborative and citation dynamics.

D8) IMDB: Very dense, heterogeneous network (142,129 nodes, 1,596,148 hyperedges, avg. degree 22.46), validating scalability on highly connected, complex structures.

These datasets span diverse densities, heterogeneity, and structures, validating our method's ability to generalise across varying graph characteristics effectively.

4. Thank you for your insightful comment. The datasets we used in our experiments were selected to ensure comparability with prior works and baseline models, which also utilized these datasets. To address your concern regarding scalability, we extended our evaluations to include two large heterogeneous datasets: IMDB (142,129 nodes, 1.59M edges) and DBLP (66,543 nodes, 274,824 edges). These results, based on accuracy metrics, are included in the revised manuscript and in Table 1.

While larger datasets like ogbn-arxiv (169K nodes) and ogbn-products (2.4M nodes) would provide further insights, our evaluations demonstrate that DistillHGNN scales efficiently to datasets of comparable complexity, such as IMDB, which is of a similar size to ogbn-arxiv. Due to GPU resource limitations and time constraints, we were unable to evaluate on these larger datasets in this submission. However, we plan to include these experiments in future work to further validate the scalability of DistillHGNN.

Thank you for your comprehensive and valuable comments, which have played a crucial role in improving the manuscript.

Weaknesses:

1. Our approach advances beyond LightHGNN through a dual knowledge transfer mechanism that ensures comprehensive preservation of high-order information:

A. Structural Knowledge Transfer:

• Teacher HGNN captures hypergraph structure through the hypergraph Laplacian (Eq. 1)
• This structural knowledge is transferred via contrastive learning using InfoNCE loss (Eq. 6), which ensures alignment between teacher embeddings $Z^t$ and student embeddings $Z^s$

B. Predictive Knowledge Transfer:

• Soft labels $Y_v^t$ from the teacher model capture class probability distributions.
• Knowledge is transferred through KL divergence in the student loss (Eq. 12), preserving the teacher's prediction patterns.

Unlike LightHGNN which relies solely on soft-target constraints, our method provides: i) Direct embedding alignment through contrastive learning ii) Explicit preservation of hypergraph structure through InfoNCE loss iii) Fine-grained knowledge transfer via both embeddings and probability distributions

2. The student model (TinyGCN) utilizes a simplified architecture that operates on node features (X) and a basic adjacency matrix $(A^s)$, enabling efficient processing of new nodes without full graph reconstruction. The single-layer aggregation $(Z^s = A^s X W^s)$ reduces computational overhead and allows incremental processing using local neighborhood information.

Additionally, the student model captures high-order relationships through the teacher’s guidance, with the learned weight matrix (W^s) facilitating efficient application to new nodes without relying on the original hypergraph structure.

Thus, our model's simplified design and efficient knowledge transfer make it suitable for dynamic environments, addressing inductive learning challenges while maintaining performance.

3. The datasets used in our experiments were selected to align with prior works and baseline models. We have added the IMDB dataset (142,129 nodes, 1.59M edges) and the DBLP dataset (66,543 nodes, 274,824 edges) to Table 1, as shown in the revised version.
4. Our analysis includes models such as LightHGNN (Feng et al., 2024) and GLNN (Zhang et al., 2021). To provide a more comprehensive evaluation, we have expanded the comparison in the revised version to include the seminal GCN model (Kipf & Welling, ICLR 2017) and KRD (Wu, ICML 2023) in Table 1.

Questions:

1. Please refer to Weakness 1.
2. Please refer to Weakness 2.
3. Please refer to Weakness 3.
4. We added the section "Visual Analysis of Knowledge Transfer" and included t-SNE visualizations comparing teacher and student embeddings in Figure 7 in the revised version of the manuscript. On the other hand, our dual-channel knowledge preservation operates as follows:

1. High-Order Knowledge Transfer via Message Passing and Contrastive Learning

• The teacher HGNN employs message passing through multiple hypergraph convolution layers ($H^1 \rightarrow H^2 \rightarrow \cdots \rightarrow H^L$), where each layer aggregates information from higher-order neighborhoods using the hypergraph incidence matrix $\mathcal{H}$.
• TinyGCN preserves this structural information through simplified message passing on the homogeneous graph ($A = \mathcal{H}W\mathcal{H}^T - D_v$).
• The contrastive loss ($\mathcal{L}_{\text{Con}}$) ensures the alignment between teacher embeddings $Z^t$ and student embeddings $Z^s$, effectively transferring the message-passing patterns.

2. Knowledge Preservation through Soft Labels

• The teacher's MLP generates soft labels ($Y^t$) that encode the high-order patterns captured through iterative message passing in HGNN layers.
• These soft labels are distilled into the student using the distillation component of $\mathcal{L}_{\text{student}}$.
• This process ensures the student captures both local and global structural information, combining supervised learning and distillation.

Together, these mechanisms complement each other: message passing and contrastive learning preserve structural relationships in the embedding space, while soft label distillation ensures the transfer of high-order predictive patterns. Figure 1 visually illustrates how these components work in unison.

Thank you for the thorough and insightful comments, which have greatly helped us improve the manuscript.

Weaknesses:

1. You are absolutely right. Our claim regarding memory efficiency is primarily based on the architectural design of our student model, which utilizes a simplified TinyGCN with a single layer. This streamlined architecture inherently reduces memory consumption by efficiently distilling high-order relational structures from the teacher model into a more compact representation. We have revised this sentence in the updated version.

2. In terms of comparative evaluation with existing distillation approaches, we have included multiple baseline comparisons. Our analysis encompasses models such as GLNN (GNN-to-MLP), as detailed in Table 1, and KRD (GNN-to-MLP), as presented in Table 2. To ensure a more comprehensive evaluation, we have expanded the comparison in the revised version to include the GCN model (Kipf & Welling, ICLR 2017) and KRD method (Wu, ICML 2023) in the updated Table 1.

Minor:

1. You are correct; "Framework 1" was intended to refer to "Figure 1." We appreciate your attention to detail and have made the necessary correction in the revised version. Thank you for pointing this out.

2. We have revised the caption for Figure 1 to be more concise and moved additional details to Section 3, as suggested. The updated caption now includes the role of soft labels in the proposed approach. Specifically, the teacher model (HGNN) generates two key outputs: (1) node embeddings $Z^t$, derived from the hypergraph Laplacian (Eq. 5), which capture high-order structural relationships, and (2) soft labels $Y_v^t$, which encode these high-order relations and are produced through the HGNN-MLP pipeline.

These soft labels serve a dual purpose: first, they are incorporated into the teacher model’s supervised loss function (as the second term) to ensure alignment between the teacher’s predictions and the ground truth when available. Second, in cases of insufficient labeled data, these soft labels function as pseudo-labels, facilitating effective training of the student model.

Questions:

1. Generally, higher accuracy in models, such as traditional GNNs and GCNs with fully connected layers, often requires increased inference time. However, in Figure 2, we compare models based on both accuracy and inference time. This comparison does not imply a direct dependency between the two. Instead, DistillHGNN and LightHGNN, strives to balance accuracy and inference time effectively. The goal of knowledge distillation models, such as ours, is to achieve this balance while improving both metrics simultaneously.

2. We appreciate your insightful observation and agree with your concern regarding the teacher model's parameter updates. To clarify, as illustrated in Figure 1, the teacher model undergoes an independent pre-training phase using the supervised loss defined in Eq. 10. During this phase, the teacher’s parameters, which include those for the HGNN and the MLP, are updated to ensure the model captures the high-order structural relationships inherent in the hypergraph.

After this pre-training phase, the teacher model’s parameters are frozen. The student model is then trained separately using its own loss function, with coordination between the teacher and student achieved via the distillation of embeddings and soft labels. This ensures the student learns from the high-order relationships distilled from the teacher while preserving the teacher’s pre-trained knowledge. To address the inconsistency noted in Algorithm 1, we have revised the pseudocode to explicitly reflect this process.

3. In the revised version, we standardized the accuracy axis range in Figure 6 to enhance clarity and facilitate comparison.

4. Thank you for the thoughtful suggestion regarding adaptive knowledge distillation. While our fixed distillation approach effectively balances performance and complexity, we acknowledge the potential of an adaptive mechanism. We opted for a fixed approach because:

   A. It ensures stable knowledge transfer between HGNN (teacher) and TinyGCN (student).
   B. Manual tuning of hyperparameters ($\gamma$, $\lambda$) provides dataset-specific adaptation.
   C. Its simplicity aligns with our goal of reducing computational complexity.