SpeAr: A Spectral Approach for Zero-Shot Node Classification

We thank the reviewer for taking the time to review our manuscript and for the valuable comments. Below is a point-by-point response to the comments.

> Response to Q1: Discussion on $k$

In Section 3.2 of the main body, the eigenvector matrix is obtained by spectral decomposition, $F^{\ast} = {[q_1, q_2, ..., q_k]}^T \in R^{N\times k}$, which serves as a novel representation of the sample. Let $z_i$ be the $i^{th}$ row of the matrix $F^{\ast}$, It turns out that $z_i$ can serve as desirable node embeddings of $x_i$. Thus, the dimension of the feature space is $k$.

The classification capability of the SpeAr algorithm is closely related to the embedding dimension $k$. Specifically, the choice of $k$ is influenced by the rank of the adjacency matrix. A larger $k$ allows the algorithm to better capture subtle differences between similar samples. In our experiments with three datasets, we chose $k$=2048 to represent a broad range of variations in the data, not just category distinctions.

> Response to Q2: The unsupervised spectral contrast loss

Unsupervised contrastive loss pre-trains the model by leveraging the inherent information within the samples. For more information on the importance of pre-training, please refer to the details in General Response. Thank you for your patience.

When constructing the unsupervised contrastive loss, each node's positive sample is defined as itself, while the negative samples are all other samples. This approach allows the model to fully utilize the intrinsic information of the data during the pre-training phase, providing more accurate feature representations for subsequent tasks.

> Response to Q3: The relationship between theory and practical performance

Given that your concerns align with Reviewer HC2X, we have provided a comprehensive elaboration on the relationship between theory and performance in General Response. We sincerely invite you to review our response in the General Response, which we believe will address your questions. We are deeply grateful for your patience and attentive consideration.

Thank the reviewer for carefully reading and for the valuable comments. We greatly appreciate the time and effort you have taken to provide such thoughtful feedback. Below is a point-by-point response to the comments.

> Response to W1: High computational complexity

Given the shared concerns expressed by Reviewer rWPM and yourself, we have formulated a comprehensive and unified response to this comment within our General Response. We respectfully request that you consult this General Response for our full explanation and clarification of this issue.

> Response to W1 and Q2: The important of pre-trained

The first phase is our pre-training phase. In the Ablation Study (Section 4.3) of the main body, we validate the model without the first phase (Model2). As shown in the following table, the Model2's results decline across three datasets.

In SpeAr, we use the pre-trained model mainly for several reasons:

• Improve the training efficiency of the second phase: In SpeAr, the first phase leverages the unsupervised contrastive loss to train the model. In the second phase, the focus is on training only the final layer of the network. This approach improves training efficiency by reducing GPU memory usage and significantly decreasing the number of gradient updates in the second stage.

• Simulate real-world workflows: In the industry, it is standard practice to first train a model using self-supervised loss on unlabeled data. In the first phase, the model can take advantage of the unlabeled data to learn useful representations from the data and enhance the model's understanding of the data.

> Response to W2 and Q3: The relationship between theory and practical performance

Given that you share the same concerns as Reviewer 7tvx, we have addressed the relationship between theory and performance comprehensively in General Response. We kindly request you to refer to the General Response for our detailed response, which we hope will clarify your doubts. Your patience is greatly appreciated.

> Response to Q1: Differences between traditional zero-shot learning (ZSL) and zero-shot node learning

While both traditional ZSL and zero-shot node classification (ZNC) aim at the recognition of unseen class samples, there are some differences:

1. Data type:

• ZSL: Images, text, or audio data is independent identical distribution.

• ZNC: Graph data, nodes form a network structure through connectivity (edges).

2. Feature representation approaches:

• ZSL: Features are usually extracted by pre-trained feature extractors (e.g. ResNet).

• ZNC: Feature representation not only considers the nodes attributes but also integrates the neighbor information of the nodes and the global graph structure.

3. Semantic migration pathway:

• ZSL: Use semantic information (e.g., word vectors, attribute vectors) to transfer semantic knowledge from seen classes to unseen ones.

• ZNC: Rely on not only semantic information but also relationships between nodes for semantic migration.

To summarize, traditional ZSL is significantly different from ZNC in terms of data types, feature representations, and semantic migration pathways.

In future work, applying the SpeAr to ZSL for image classification tasks is feasible. The critical aspect is constructing the neighbor relationships between samples, which can be achieved through feature similarity or image augmentation strategies.

> Response to Q4: The choice of the hyperparameter $s$

For the choice of the hyperparameter $s$, it is set to 1000 across all three datasets. In SpeAr, we found that if $s$ is set too small, its effect on prototype updates is not significant. As shown in the table below, when $s$=100, the results across the three datasets are poor. Based on the experimental results from all three datasets, we found that $s$=1000 achieved better performance consistently.

We are immensely grateful to you for recognizing the reasonability and presentation of our work. Your valuable suggestions inspire us to improve our work further.

If you think the following response addresses your concerns, we would appreciate it if you could kindly consider raising the score.

> Replay to W1: Differences from the spectral contrastive loss in [29]

Our loss is designed based on the data and tasks, which is quite different from the loss in literature [29], as shown in the Table.

Utilizing the spectral contrastive loss from [29] in ZNC is hindered by data and objectives. [29] targets image-level representations, establishing sample relationships via augmentation. Lacking labels and natural node adjacencies, [29] cannot effectively uncover positive correlations among similar nodes or negative ones between dissimilar nodes.

It is important to note that our innovations are:

1. Combine the spectral contrastive learning with the learnable class prototypes to discover the implicit clustering information and realize the semantic migration, thus further alleviating the bias problem.
2. Make full use of labeling and adjacency to design node spectral contrastive loss mining the implicit clustering information in unlabeled nodes (Reviewer HC2X also notes this).
3. Theoretically, we derive an approximate relationship between the obtained node embeddings and the feature vectors obtained from spectral decomposition.

> Replay to W2 & Q1: The efficiency and training time

Given that your concerns align with Reviewer HC2X, we have responded to this issue in the General Response. We sincerely appreciate your time and patience, and kindly encourage you to navigate to that section for our comprehensive response.

> Replay to W3: The sensitivity of the hyperparameters

We have set guidelines for parameter selection that efficiently guide in choosing suitable hyperparameters and facilitate swift model tuning for new data.

Hyperparameters $\alpha$ and $\beta$

We analyze the $\alpha$ and $\beta$ across three datasets (Figure 1 in the additional rebuttal PDF). The $\alpha$ and $\beta$ regulate the contribution of labeled and unlabeled samples in overall loss. The results reveal a notable trend: when the value of $\beta$ is larger than $\alpha$, the model generally exhibits better performance, with this advantage being particularly prominent when $\beta = 1$.

The conclusions meet our expectations. A larger $\beta$ can reduce overfitting to seen classes and better exploit the clustering information embedded in unlabeled nodes. Thus, the parameters need to satisfy $\beta > \alpha$. This suggests that the $\alpha$ and $\beta$ are not sensitive to some extent.

Hyperparameters $q$

When a sample's pseudo-label probability exceeds the threshold $q$, it updates the class prototypes. The choice of $q$ depends on model and dataset: low $q$ risks noisy pseudo-labels, degrading performance; high $q$ ensures accuracy but limits unlabeled data utilization [1-3].

Fortunately, SpeAr achieves satisfactory performance across datasets with $q$=0.5 (Figure 2(c) in Section 4.4). Even on Cora, our result (55.49) at $q$=0.5 significantly surpasses SOTA (48.43), validating the effectiveness of SpeAr. The strategies for handling new datasets are:

1. Set $q$=0.5, assigning lower weights to samples during prototype updates to mitigate noise.

2. Select $q$ using the validation set (Figure 2 in the additional rebuttal PDF). Across three datasets, as $q$ varies, the accuracy changes in both the test and validation sets are roughly consistent, confirming the strategy's effectiveness and reliability. This will be discussed in the modified version.

3. In the future, we will focus on advanced threshold selection, including adapting thresholds to class difficulty and dynamically updating them based on model learning.

[1] Fixmatch: Simplifying semi-supervised learning with consistency and confidence

[2] Boosting semi-supervised learning by exploiting all unlabeled data

[3] SemiReward: A General Reward Model for Semi-supervised Learning

> Replay to W4: Inadequate experimental settings

We give all the hyperparameters involved in our SpeAr (Table 1 in the additional rebuttal PDF).

> Replay to Q2: Caution in using seen classes to prevent overfitting

We accounted for this in the initial manuscript (Section 4.4) by adjusting the weights of $\alpha$ and $\beta$ to further alleviate overfitting issues.

First, the seen class information serves as a pivotal semantic bridge during the training phase, facilitating SpeAr's ability to capture and transmit the deep semantic ties between classes, enhancing the model's capacity for generalization towards unseen classes.

Second, we adjusted the weights of seen and unseen classes within the spectral contrastive loss to mitigate the overfitting to seen classes, setting $\beta$ to be greater than $\alpha$. The adjustment also improves the model's adaptability to unseen classes.

Recognizing that our manuscript's lack of detailed explanation may have led to confusion, we will add a detailed discussion in the revised version.