HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs

[1]: Dongjin Lee and Kijung Shin. I’m me, we’re us, and i’m us: Tri-directional contrastive learning on hypergraphs. In AAAI, 2023.

[2]: Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Augmentations in hypergraph contrastive learning: Fabricated and generative. In NeurIPS, 2022.

[3]: Nate Veldt, Austin R Benson, and Jon Kleinberg. Combinatorial characterizations and impossibilities for higher-order homophily. Science Advances, 9(1):eabq3200, 2023.

[4]: Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In IJCAI, 2021.

Comment

I noticed that in section 5.1, the article uses a setting of 1% of the training set, which is different from most articles. Most other papers use fixed division or 5/10 nodes per category. I'm curious about what the considerations are for such an experimental setup?

Response

• Experiment Q2.1: In the reviewer’s suggested setting, HypeBoy outperforms two SOTA hypergraph SSL methods (TriCL and HyperGCL).
  • Since a fixed division split that is widely used for hypergraph node classification is unknown, we have employed the second setting: 5/10 training nodes per category.
  • The results are shown below, where each method employs UniGCNII [4] as a backbone encoder.
  • We have added this result to the revised manuscript as Table 11.
• Clarification Q2.2: Moreover, we would like to clarify that our used setting is equivalent to the setting of Wei et al., 2022. [2].

Additional Data Splits Experimental Results

5 training nodes

10 training nodes

Comment

In BASIC SETTING, one of the assumptions is that the homophily ratio of each hyperedge is in [0.5, 1]. In graphs, the homophily rate is defined as the proportion of intra-class edges to all edges. So how does this definition apply to the hypergraph? Do all datasets in the experiments satisfy this assumption?

Response Summary

• Our theoretical analyses hold for both general-sense homophily- and non-homophily (i.e. homophily means that the same-class nodes tend to interact with each other).
• Empirically, in both homophilic and non-homophilic hypergraphs, our method (HypeBoy) outperforms other SSL methods.

Response

• Clarification Q1.1 [Theory]:
  • We are sorry for the confusion. The additional descriptions for $\mathscr{P}$ are provided in Clarification for Homophily Ratio.
  • We again clarify that our theorem is valid at any $\mathscr{P} \in [0,1]$. In other words, Theorem 1 and 2 hold for both homophilic and non-homophilic hypergraphs.
• Clarification Q1.2 [General-sense Homophily Measure]:
  • We have used one of the widely used hypergraph homophily analysis methods, proposed by Veldt et al. 2023 [3].
  • The homophily ratio the reviewer has mentioned can be directly extended to hypergraphs as the proportion of hyperedges that consist of the same-class nodes to all hyperedges.
  • However, this has limitations in measuring the extent of homophily within each hyperedge.
    • For instance, consider two hyperedges $e_{i}$ and $e_{j}$ whose constituent node labels are $\{{0,0,0,0,0,1}\}$ and $\{{0,0,0,1,1,1}\}$, respectively.
    • Although their label-mixed-extents significantly differ, both hyperedges are counted as non-same-class-node hyperedges.
    • In this manner, this extension may fail to measure the higher-order homophily in a hypergraph.
  • Notably, the homophily analysis method (Veldt et al. 2023) generalizes the homophily metric the reviewer has mentioned (i.e., the proportion of intra-class edges to all edges).
• Experiment Q1.3 [Homophily Levels in Hypergraphs]:
  • Since the homophily analysis is based on visualization analysis, we could not report a single value that represents the homophily of a dataset. Instead, analysis should be conducted by each hyperedge size (each hyperedge size has a separate plot). For each size plot, the line color represents the node class.
    • An upward trend line indicates the corresponding class nodes exhibit higher-order homophilic characteristics in the corresponding size hyperedges.
    • A downward trend line indicates the corresponding class nodes exhibit higher-order non-homophilic characteristics in the corresponding size hyperedges.
  • We analyze three benchmark datasets: Cora, IMDB, and AMiner. The results are in Figure 6 of the revised manuscript (Appendix). Notably, these three datasets exhibit different homophilic characteristics. Details regarding the homophily analysis are as follows.
    • Cora (Homophilic): Most classes show an upward trend.
    • IMDB (Non-Homophilic): Most classes show a downward trend.
    • AMiner (Mixed): Some classes show an upward trend and some classes show a downward trend.
• Experiment Q1.4 [Performance on Heterophilic Hypergraph]:
  • In short, HypeBoy outperforms other baseline methods in these three datasets: homophilic, non-homophilic, and partially homophilic hypergraphs (Table 1 of the main paper).
  • Thus, we have verified that the empirical efficacy of HypeBoy is not limited to homophilic hypergraphs.
  • This analysis is added to the revised manuscript.

Comment

One of the main contributions of the paper is hyperedge filling, but using only hyperedge filling (v1) in the ablation experiment is the worst among v1-v4. The effectiveness of using hyperedge filling alone is questionable, or it needs to be bound to feature reconstruction and projection heads.

Response Summary

• We show that the hyperedge filling task is a better SSL task compared to other SSL tasks, including feature reconstruction and contrastive learning.
• We clarify that it is common to employ additional parameters (e.g., projection heads) to complement an SSL task, and our competing SSL methods also utilize such additional parameters.

Response

• We clarify the relationship between additional parameters (e.g. projection heads) and SSL methods:
  • Clarification W2.1: Projection heads are widely used tools to complement SSL methods, enhancing their performances.
  • Clarification W2.2: Typical SSL pretext tasks, contrastive learning and feature reconstruction, also require additional parameters such as projection heads, decoders, and view generators.
• Under such context, we have conducted two additional experiments to address the reviewer’s concerns:
  • We compared the hyperedge filling task with other SSL tasks under two settings: (1) encoder only, i.e., without additional parameters; (2) encoder + additional parameters. Under both settings, hyperedge filling is more effective than other SSL tasks.
  • Experiment W2.3 [Encoder-Only]:
    • Summary: When an encoder alone is employed, the hyperedge filling task is more effective than other hypergraph SSL tasks.
    • Setting: We compare various SSL methods by removing their additional parameters.
      • Hyperedge filling: It employs an encoder without projection heads (equivalent to V1 in the main paper).
      • Feature reconstruction: We remove a decoder from the feature reconstruction method. Specifically, we make the embedding dimension of the encoder the same as the node feature dimension (other parts are the same as our feature reconstruction scheme).
      • Contrastive learning: We remove projection heads from TriCL [1] (contrastive loss and augmentation steps are equivalent to TriCL).
    • Outcome details: As shown in the below table, hyperedge filling without any parameters (V1) outperformed (1) feature reconstruction (V3) or (2) contrastive learning methods.
    • Revision: We have added the results to the revised manuscript as Table 8.
  • Experiment W2.4 [Encoder + Additional Parameters]:
    • Summary: When additional parameters are also employed, the hyperedge filling task is more effective than other hypergraph SSL tasks.
    • Setting: We compare various SSL methods that employ additional parameters.
      • Hyperedge filling: This method employs an encoder with projection heads (equivalent to V2 in the main paper).
      • Feature reconstruction: We employ both encoder and decoder (equivalent to V3 in the main paper).
      • Contrastive learning: We employ TriCL and HyperGCL [2].
    • Outcome details: By comparing the performance of hyperedge filling + projection head (V2) with that of V3, TriCL, and HyperGCL, we have verified that V2 outperforms other methods in terms of the average rank (this result can be obtained from Table 1 and 3).

SSL methods W/O additional parameters

Comment

In the hypothesis, the premise of the article is that each hyperedge contains most of the same category vertices. However, there are hyperedges that do not satisfy this situation. For example, a hyperedge of size 10 contains 5 nodes of category A and 5 category B.

Response

• We apologize for the confusion. The detailed descriptions regarding $\mathscr{P}$ are provided in “Clarification for Homophily Ratio”.
• Clarification W1.1: Our theoretical analysis covers both general-sense homophily and non-homophily (i.e. homophily means that the same-class nodes tend to interact with each other).
  • Both Theorem 1 and 2 hold for any $\mathscr{P} \in [0, 1]$, i.e., both homophilic and non-homophilic cases.
  • With $\mathscr{P} = 0.5$, each hyperedge is expected to contain half class-0 nodes and class-1 nodes.
• Experiment W1.2: To demonstrate HypeBoy’s efficacy when the homophily assumption of input data is violated, we conduct an additional experiment, described in the tables below.
  • Results: HypeBoy is the most robust to decreasing homophily. Its performance gap to the SOTA baselines tends to increase as the homophily assumption of the input data is increasingly violated.
  • The experimental details are as follows:
    • We randomly choose two hyperedges ($e_{1}$ and $e_{2}$), and choose one node each from the chosen hyperedges ($v_{1} \in e_{1}$ and $v_{2}\in e_{2}$).
    • Then, we swap two chosen nodes: $e^\prime_1 = (e_1 \setminus \{v_1\}) \cup \{v_2\}$ and $e^\prime_2 = (e_2 \setminus \{v_2\}) \cup \{v_1\}$, then $e_{1} = e^\prime_{1}$ and $e_{2} = e^\prime_{2}$.
    • We repeat this process. With more swapping, a hypergraph is expected to get more non-homophilic.
  • The results are added in the revised manuscript as Table 10.
• Argument W1.3: HypeBoy is more effective than its competitors in less homophilic hypergraphs.
  • The proposed method (HypeBoy) outperforms the baseline methods in non-homophilic benchmark datasets. See more details in the response for Q1.

Node Swapping Experimental Results

Cora (Value in a bracelet indicates the performance gap from that of the original hypergraph)

Citeseer (Value in a bracelet indicates the performance gap from that of the original hypergraph)

• Clarification 1. Our theoretical analysis covers both homophily and non-homophily scenarios.
  • Specifically, our theorems (Theorems 1 and 2) are valid in both homophilic and non-homophilic cases.
    • The (expected) general-sense homophily (i.e. the same class nodes tend to interact with each other) is in a symmetric U-curve relation w.r.t. $\mathscr{P} \in [0,1]$.
      • Homophily decreases at $\mathscr{P} \in [0, 0.5]$ and increases at $\mathscr{P} \in [0.5, 1]$.
      • Homophily is maximized at either $\mathscr{P} =1$ or$\mathscr{P} = 0$ and minimized at $\mathscr{P} = 0.5$.
    • Our theorems (Theorems 1 and 2) hold for any $\mathscr{P} \in [0,1]$, i.e., both homophilic and non-homophilic cases.
• Clarification 2. $\mathscr{P}$ is a parameter that controls the number of particular class nodes in a hyperedge.
  • Specifically, homophily does not monotonically increase as $\mathscr{P}$ increases (see Clarification 1 above).
  • We apologize for the confusion seemingly caused by our verbal expression of $\mathscr{P}$ (we used the term “homophily ratio”).
  • We have renamed the $\mathscr{P}$ as “affinity parameter” and modified it in the revised manuscript with additional descriptions.
  • Details regarding $\mathscr{P}$ are described below:
    • In our theoretical analysis, we assume two types of hyperedges (Type 0 and Type 1).
    • (Homophilic) As $\mathscr{P} \rightarrow 1$, Type 0 hyperedges tend to be filled with class-0 nodes, and Type 1 hyperedges are filled with class-1 nodes.
    • (Homophilic) As $\mathscr{P} \rightarrow 0$, Type 0 hyperedges tend to be filled with class-1 nodes, and Type 0 hyperedges are filled with class-0 nodes.
    • (Non-Homophilic) As $\mathscr{P} \rightarrow 0.5$, for each hyperedge, half of it tends to be filled with class-1 nodes, and the rest half tends to be filled with class-0 nodes.
  • Our data model is symmetric about $\mathscr{P} = 0.5$. In other words, $\mathscr{P}$ and $1 - \mathscr{P}$ are expected to produce equally homophilic hypergraphs.
    • Considering the entire hyperedges, the tendency of node classes being mixed within hyperedges is equivalent at hypergraphs that are generated with $\mathscr{P}$ and those with $1-\mathscr{P}$.

Dear Reviewer EPYj,

We appreciate your constructive review. We provide our responses to each of your comments below. First, however, please let us make clarifications about the homophily ratio, which would serve to better clarify our responses.

[1]: Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI, 2019.

[2]: Dongjin Lee and Kijung Shin. I’m me, we’re us, and i’m us: Tri-directional contrastive learning on hypergraphs. In AAAI, 2023.

[3]: Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In ICLR, 2022.

[4]: Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In IJCAI, 2021.

[5]: Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. In ICML Workshop on Graph Representation Learning and Beyond (GRL+), 2020.

[6]: Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.

Comment

In the hyperedge filling process (F2), the representation is updated with a gradient w.r.t. the hyperedge filling loss and a step size. Per my reading of the proof, the value of (only need to be greater than 0) seems to not affect the result, while in practice this might not be true. Do I miss some points here?

Response

• Clarification W2.1: Theorem 1 only states that the updated embedding is more beneficial and does not discuss the extent of benefit, which is expected to be influenced by the step size, as the reviewer pointed out.
• Clarification W2.2: In practice, as the reviewer has mentioned, the efficacy of the hyperedge filling task is influenced by the step size. On the other hand, with a fixed step size of 0.001, HypeBoy outperforms existing hypergraph SSL methods in most of the datasets (in all our experiments, we have fixed the step size of 0.001, while that for other methods are tuned. Details are in Appendix).
• Argument W2.3: Thus, we conclude that HypeBoy is relatively robust to the step size.
• Clarification W2.4: Apart from the theoretical results, we delineate some possible reasons why step size may affect the node classification performance
  • Real-world datasets may not (perfectly) align with our data assumptions that are used in theoretical analysis.
  • We use a hypergraph neural network to obtain representations of nodes in practice, while we have assumed each representation as a free variable that is directly being updated.

Comment

In the hyperedge filling analysis, it seems to not relate to the neural network architecture. How do authors think the choice of the hypergraph neural networks would affect the performance?

Response

• Argument W1.1: We have verified that a neural network that preserves the higher-order information of each hyperedge aligns well with HypeBoy.
• Empirical Support W1.2: We have employed four different encoders as the backbone encoders of HypeBoy. Among the four, the encoder that uses higher-order information the least, HGNN [1], shows the worst performance.
  • Specifically, the encoders are HGNN [1], MeanPoolingConv [2], SetGNN [3], and UniGCNII [4].
  • A noticeable difference between HGNN and the other encoders is their use of input data structure: HGNN transforms a hypergraph into a graph via clique-expansion and performs graph convolution, while the others conduct message passing on the hypergraph structure itself (details are described in Section 2.1 of the main paper).
  • As discussed by Dong et al., [5], clique expansion may cause significant information loss in higher-order interactions, potentially causing suboptimal performance of a machine learning model.
  • To effectively solve the hyperedge filling task, an encoder should be capable of learning higher-order interactions, and such a requirement may cause a suboptimal performance of HGNN.

Encoder Experiments

Dear Reviewer 4MNM,

We appreciate your generous review. We provide our responses to each of your comments below.

I acknowledge the response and would like to increase my positive rate 6 --> 8.

Dear Reviewer 3LGX,

We deeply appreciate your constructive review. Below, we provide our response to each of your comments. Our empirical outcomes have become more solid, thanks to your comments.

Comment

Given the big improvement brought by the augmentation (masking) scheme (Table 6 in appendix), I am curious to know if the other baseline methods benefit from a similar augmentation step.

Response

• Clarification W4.1: HyperGCL and TriCL (the SOTA hypergraph SSL baseline methods) also employ augmentation strategies, and their results were obtained with the augmentations.
  • Specifically, both HyperGCL and TriCL are contrastive learning methods that create hypergraph views by augmenting input hypergraphs.
  • To this end, they utilize the below methods:
    • HyperGCL: This method generates augmented views of a hypergraph by using a hypergraph VAE model. The hypergraph VAE model is being trained with contrastive loss.
    • TriCL: This method masks certain columns of the input node feature matrix and drops a certain number of nodes from each hyperedge.
• Argument W4.2: Thus, we emphasize the fairness of our experiments and the superiority of our proposed model and self-supervision loss.

Comment

Minor: It would be useful to include the UNIGCNII baseline (without any of the 3 component) as a line in Table 3

Response

• We have revised Table 3 based on your comment: UniGCNII [5] is added as an additional baseline in Table 3.
• As we originally concluded, we find that each component of HypeBoy contributes to its performance gain.

Updated Table 3 (The first line “NA” is added).

The ablation study with four variants of HypeBoy on node classification under the fine-tuning protocol. F.R., H.F., and P.H. denote Feature Reconstruction, Hyperedge Filling, and Projection Heads, respectively. A.R. denotes the average ranking among all methods. NA denotes no pretraining. HypeBoy outperforms others in most datasets, justifying each of its components.

Comment

While the authors did a good job in explaining the intuition behind this, it is somewhat discouraging to observe that the suggested approach performs noticeably worse when the feature reconstruction warmup is omitted (Table 3).

Response

• We make the following clarifications:
• Clarification W2.1: Our method outperforms two SOTA hypergraph SSL methods, even without feature reconstruction warmup.
  • In terms of the average ranks, HypeBoy variant V2 (no feature reconstruction) outperforms TriCL and HyperGCL (the SOTA hypergraph SSL methods).
  • This can be derived by comparing Tables 1 and 3 of the main paper.
• Clarification W2.2: Moreover, the hyperedge filling task is more effective than feature reconstruction.
  • In terms of the average ranks, V2 (hyperedge filling with projection heads) outperforms V3 (only feature reconstruction).
• Clarification W2.3: Lastly, feature reconstruction on hypergraphs is also our contribution, since we are the first to employ the feature reconstruction scheme on hypergraphs, to our best knowledge.

Comment

All the SSL-based results presented in the paper uses UniGCNII as a backbone. The authors motivate this decision by saying that this combinations achieves best overall performances. While I consider the comparison fair (since all the reported baselines uses UniGCNII as well), it is essential to conduct experiments that demonstrate the method's advantages when applied with various backbones. This additional experiments would clearly demonstrate the advantages of the proposed method.

Response

• We additionally have employed three hypergraph encoders used in SOTA hypergraph SSL methods (TriCL [1] and HyperGCL [2]).
  • Additional encoders: HGNN [3], MeanPoolingConv [1], and SetGNN [4]
• Our method (HypeBoy) outperformed all SOTA hypergraph SSL methods in all the used encoders. The results are presented in the below tables.
  • Details: Compared to the 2nd-best SSL method (TriCL), HypeBoy achieves up to 4.2% better accuracy in all the encoder settings.
• We have added these results in Table 7 of our revised manuscript.

Encoder Experiments

Encoder: HGNN [3], used in TriCL [1].

Encoder: MeanPoolingConv [1], used in TriCL [1].

Encoder: SetGNN [4], used in HyperGCL [2].

[1]: Dongjin Lee and Kijung Shin. I’m me, we’re us, and i’m us: Tri-directional contrastive learning on hypergraphs. In AAAI, 2023.

[2]: Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Augmentations in hypergraph contrastive learning: Fabricated and generative. In NeurIPS, 2022.

[3]: Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI, 2019

[4]: Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In ICLR, 2022.

[5]: Jing Huang and Jie Yang. Unignn: a unified framework for graph and hypergraph neural networks. In IJCAI, 2021.

Dear Authors,

Thank you for carefully addressing the review.

Before the rebuttal, my main concern regards the absence of experiments involving multiple backbones. The additional experiments included in the response demonstrate that the proposed method yields significant improvements irrespective of the hypergraph processing method. I believe that this is an important empirical contribution, which strengthen and validates the proposed approach.

In light of this, I decided to increases my score to 6: marginally above the acceptance threshold.

Best, Reviewer 3LGX

Comment

Could you elaborate on the real-world applications of the hypergraph filling task? How is this task practically relevant?

Response

• As described in the Overall Clarification, our primary objective is to empower a hypergraph neural network to better solve the node classification task. That is, we employed the proposed hyperedge filling task as a means to achieve this goal, rather than considering it as the final objective.
• While hyperedge filling is not our ultimate goal, hyperedge filling itself has potential applications described below. We have added these potential applications in our revised manuscript.
  • Email recipient recommendation
    • Given recipients of an email, recommend a user likely to be added as a recipient of the email.
    • Nodes in a hypergraph indicate a user and each hyperedge indicates an email, consisting of a sender, recipients, and CCs.
  • Item recommendation
    • Given a shopping cart list, recommend an item to be co-purchased with listed items.
    • Nodes in a hypergraph indicate an item and each hyperedge contains a set of co-purchased items.

Comment

It's not straightforward to extend edge reconstruction methods to hyperedges in SSL. Considering this challenge, why did the author choose to employ SSL for hypergraphs without addressing the embeddings of hyperedges explicitly.

Response

• We make the following clarifications:
  • Clarification Q1.1: Explicit hyperedge embeddings are not required for both hyperedge filling and node classification, which are the focus of this paper.
    • Hyperedge filling task
      • This task aims to fill a correct node to a given subset of a hyperedge. To this end, we employ representations of a node and a subset, whose steps are described in Section 4.2.
      • Thus, we do not need hyperedge embeddings to perform hyperedge filling.
    • Node classification task
      • This task aims to correctly classify a node. To this end, we employ a node representation.
      • Thus, we do not need hyperedge embeddings to perform node classification.
  • Experiment Q1.2: Our proposed hyperedge filling method outperforms the hyperedge reconstruction method, which is a direct extension of graph edge reconstruction.
    • Hypergraph reconstruction method details are as follows:
      • For each SSL training epoch, we create negative-sample hyperedges by using the size-negative-sampling strategy, which is described in Appendix D.3.
      • Then, we encourage an encoder to maximize the probability of the real hyperedges, while minimizing that of the negative-sample hyperedges.
      • To obtain the probability of a hyperedge, we (1) obtain a hyperedge embedding via sum-pooling of embeddings of nodes that are included in the corresponding hyperedge and (2) feed the hyperedge embedding to the MLP classifier.
    • We have added this result to the revised manuscript as Table 9.
• Under such context, we make the following argument:
  • Argument Q1.3: The superiority of the hyperedge filling task, which we have theoretically shown the effectiveness on the node classification, over the hyperedge reconstruction method is evidenced by the results in (Experiment Q1.2).

Hyperedge Reconstruction SSL Method

Comment

The author utilizes UniGCNII[2] as an encoder for HYPERBOY, and primarily focuses on homogeneous hypergraphs. However, it's important to note that heterogeneous hypergraphs are also prevalent, and the embedding method for hyperedges is not discussed.

Response

• To address your concerns, we conducted further experiments about heterogeneous hypergraphs. We summarize the results below:
  • Experiment W3.1: In a heterogeneous hypergraph dataset, we have verified that HypeBoy outperforms two SOTA hypergraph SSL methods.
    • Dataset: We found one heterogeneous hypergraph dataset (the ACM dataset [7]) that meets the following criteria: (1) exhibiting heterogeneity, (2) having node attributes and labels, and (3) being publicly accessible.
    • Results: HypeBoy achieves the best performance among the SOTA SSL baselines. The results are provided in the below table. We have added this result to the revised manuscript as Table 12.
• Further, we make the following clarification w.r.t. hyperedge embeddings:
  • Clarification W3.2: Our primary goal is to obtain node representations for node classification. In this process, we do not explicitly obtain or utilize hyperedge embeddings.
  • Clarification W3.3: Potentially, however, we can obtain good hyperedge embedding by appropriately employing node representations obtained from a hypergraph neural network.

Results on Heterogeneous Hypergraph (ACM)

Comment

The author devises a SSL strategy for hypergraphs by utilizing existing encoders and decoders such as UniGCNII, HNN, and MLP, without introducing any novel model designs. The concept of the projection head for hypergraph encoding is inspired by Deep Sets[1].

Response

• We make the following clarifications:
  • Clarification W2.1: We emphasize that our main contribution lies in devising a pretext task for hypergraph SSL, as described in Section 2.1 of the main paper, not proposing a novel neural network architecture for hypergraphs.
  • Clarification W2.2: Moreover, as evidenced in the below table, the proposed hyperedge filling task can empower various hypergraph neural networks, not only UniGCNII, in the node classification task.
    • We have additionally employed three hypergraph neural networks, which were used as backbone encoders of SOTA hypergraph SSL methods (TriCL [3] and HyperGCL [4]).
    • As shown below, HypeBoy outperforms other SSL methods in most of the settings.
• As such, we argue the following:
  • Argument W2.3: Devising a pretext task, hyperedge filling, that can empower a wide range of hypergraph neural networks is our significant contribution.
    • Furthermore, given HypeBoy’s potential applicability to a broad spectrum of hypergraph neural networks, our contribution is on par with introducing a novel neural network architecture.
  • Argument W2.4: Devising an SSL method (consisting of augmentation, encoding, and loss function) that satisfies the desired properties (Property 1 - 3 in Section 4 of the main paper) is our non-trivial contribution.
  • Argument W2.5: Thus, our contributions are valid even if we employ existing neural networks as the hypergraph encoders.

Additional encoder experimental results

Encoder: HGNN [5], used as the encoder for TriCL.

Encoder: MeanPoolingConv [3], used as the encoder for TriCL.

Encoder: SetGNN [6], used as the encoder for HyperGCL.

Comment

The rationale behind employing generative SSL for hypergraph representation is not convincingly established, and it confronts several challenges, including dimensional collapse.

Response

• As described in our Overall Clarification, we aimed to better solve the node classification task by empowering a hypergraph neural network with the proposed hyperedge filling task.
• Hyperedge filling helps node classification (Sec. 3): Theoretically, solving the proposed SSL pretext task, hyperedge filling, can improve node classification performance. Moreover, we have experimentally demonstrated that our method HypeBoy outperforms other baseline methods.
• Findings in other domains (Sec. 1): The effectiveness of generative SSL has been demonstrated in many downstream tasks for a range of domains [1, 2].
• Utility in label-scarce scenarios (Sec. 1): Efficacy of (semi-)supervised learning (e.g., by directly aiming at node classification) is limited when available labels are scarce. Generative SSL tackles such a label-scarcity issue by providing abundant labels from the hypergraph structure itself.
  • We experimentally validate this with our experiments in Table 1. See higher average ranks of SSL methods over the (Semi-)Supervised ones.

• We clarify that one of the primary goals of Self-Supervised Learning (SSL) is to empower a machine learning model to better solve some downstream tasks.
• To this end, the majority of SSL methods solve pretext tasks, which are distinct from the target downstream tasks.
• Our primary contribution is devising a pretext task for hypergraph representation learning, called the hyperedge filling task.
  • The goal of our pretext task: By solving the hyperedge filling task, we mainly aim to better solve the node classification task, which is one of the most popular downstream tasks in the hypergraph representation learning field.
  • Theoretical connection: While our hyperedge filling task is seemingly not related to node classification, we have theoretically analyzed that solving the hyperedge filling task is beneficial for node classification, as described in Section 3.2 of the main paper.

Dear Reviewer ctQb,

We express our gratitude for your considerate review. We summarize our response to each of your comments below. In doing so, please let us first make an overarching clarification.

[1]: Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.

[2]: Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD, 2022.

[3]: Dongjin Lee and Kijung Shin. I’m me, we’re us, and i’m us: Tri-directional contrastive learning on hypergraphs. In AAAI, 2023.

[4]: Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Augmentations in hypergraph contrastive learning: Fabricated and generative. In NeurIPS, 2022.

[5]: Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI, 2019.

[6]: Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are allset: A multiset function framework for hypergraph neural networks. In ICLR, 2022.

[7]: Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In WWW, 2019.

Comment

The author uses Gaussian distribution, Bernoulli sampling, and binomial distribution in the method. Could you explain the reasoning behind these choices and how they compare to more conventional methods like attention strategies and neural network approaches?

Response

• We clarify that the mentioned conditions are only assumed for the simplicity of our theoretical analysis (Section 3.2), and they are not utilized in our design of HypeBoy, which is our proposed SSL method.
• We have clarified this in our revised manuscript.