One Prompt Fits All: Universal Graph Adaptation for Pretrained Models

Thank you for your constructive feedback; we appreciate your insights and have addressed your comments below.

To W1 & Q2:

We thank the reviewer for the thoughtful comments.

While techniques like kNN and bootstrapping have been used in GSL and SSL, our work is different in how we apply them to graph prompt learning, with the goal of unleashing the capabilities of pretrained GNN models. The kNN algorithm is consistent with the homophily assumption underlying widely adopted pre-trained GNN models. By constructing a similarity-based graph, kNN enables the input data to better align with these models, allowing information from similar nodes to be aggregated, thereby enabling more effective adaptation. This design aligns well with our core perspective: graph prompt learning should focus on unleashing the capability of pretrained models. Compared to feature-based or connection-based prompt injection methods, kNN provides a clear and structure-based method that help the pretrained GNN models work more effectively.

• For random topology: Randomly linked graphs may lack meaningful structure, hindering the pretrained model's ability to capture node similarity or semantic relationships. This severely impacts the performance of downstream tasks, with an average accuracy drop of over 25%. Detailed experiments are provided in the response to weakness 3.

• For learnable topology: While the idea of end-to-end learnable topology is appealing, it typically requires complex design and greater computational overhead to ensure the effectiveness of the learned structure. We agree that this represents a promising direction for future research, and we thank you for highlighting this point.

To W2:

We categorize existing graph prompt methods into three main types according to their mechanisms: input-level, layer-wise, and representation-level. We want to analyze these various approaches to gain a deeper understanding of the underlying prompt mechanism.

Regarding representation-level methods, our analysis demonstrates that they are equivalent to a classifier. This suggests that their mechanism contradicts the core advantages of prompting and fails to leverage the benefits of prompts, as similar results can be achieved with a classifier alone. As for layer-wise methods, due to their inherent design complexity and strong reliance on the internal representations of pre-trained models, they are not suitable for black-box pre-trained GNNs. Therefore, we do not consider these methods in our work.

In contrast, input-level methods avoid the limitations and preserve the core advantages of prompting without requiring access to model internals. Thus, they are the most promising among the three categories for our setting. We thus design an input-level prompt mechanism. Through this process, we get our core perspective: "Graph prompt learning should focus on unleashing the capability of pretrained models, and the classifier adapts to downstream scenarios.", which clearly defines the distinct roles and mechanisms of the two crucial downstream components: the prompt and the classifier.

To W3:

We appreciate this suggestion. Since both components are essential and cannot be simply removed, we conduct replacement experiments: (1) replacing kNN with random topology, (2) replacing bootstrapp with simple addition of original and prompt graph, (3) discarding original graph totally. 1-shot results as shown in the table below:

From the table, we can find that Random_Topo maintains some of effectiveness on homophilic datasets while showing reduced performance on heterophilic ones. Conversely, for Simple Addition and Discarding Topology, heterophilic datasets still retain some performance. However, performance on homophilic datasets drops significantly, as their original structure is crucial for classification. This table will be included in the revised versions.

To W4:

We appreciate you raising this point. Since our prompt topology is built on node features, it is sensitive to feature noise, which can lead to distorted graph structures. When both features and topology are misaligned with the pre-trained model, our method faces challenges in solving this problem. The 1-shot learning results under varying levels of Gaussian noise are summarized in the following table:

We observe that a 0.01 noise level shows minor augmentation, maintaining accuracy in some datasets (e.g. PubMed, Wisconsin and Actor). However, 0.05 noise begins to impact performance, and 0.20 noise significantly degrades accuracy across most datasets, with an average accuracy drop of over 30%. This analysis will be incorporated into future versions.

For the extreme heterophily cases, our method can address these situations. Our experiments demonstrate that our approach works effectively on different heterophilic datasets. Conversely, when the alignment between upstream and downstream data is already relatively high (e.g. both training and testing use homophilic datasets), our improvement is less pronounced. This discussion will be incorporated into future versions.

To Q1:

Thank you for the suggestion. We have additionally run experiments on the large-scale heterophilic dataset Arxiv-year (169,343 nodes and 1,166,243 edges) as a supplement. Here, we use a simplified kNN by randomly sampling 1000 nodes, then connecting each node to its top-k most similar sampled nodes. We test three pretrain strategies under 5-shot setting, comparing with fine-tuning. The accuracy and computational cost are shown in the table below.

As shown in the table, our method incurs minimal preprocessing time and only a slight increase in training time per epoch, with small epoch counts (typically less than 500). This demonstrates that our approach is scalable to large graphs. This table will be included in the future versions.

To Q3:

Thank you for your thoughtful consideration. Our method is designed to be agnostic to the choice of pretrained GNN. To demonstrate this, we evaluate UniPrompt on diverse pretrained backbones, including DGI, GRACE, GraphMAE and the cross-domain model FUG. The consistent performance across these settings supports the generality of our approach.

That said, we acknowledge a limitation: most pretrained models we use are trained on homophilic datasets, and our similarity-based prompting mechanism naturally aligns with such inductive biases. If pretrained GNN models or foundation models were trained exclusively on heterophilic graphs, the effectiveness of our current strategy may vary. However, this does not conflict with current practice, nearly all pretrained GNN models and foundation models are trained on homophilic benchmarks. We intend to adapt our work to heterophilic settings as future work.

To Q4:

Thank you for your question, we provide hyperparameter analysis to $\tau$ and $k$ in Figure 4. In general, smaller $\tau$ perform better on heterophilic graphs while larger $\tau$ work better on homophilic graphs. This is consistent with the characteristics of real-world datasets, where adding more homophilic information to heterophilic datasets can further leverage topological information to improve performance. A similar principle applies to $k$, smaller $k$ works well when pretrained models are trained on homophilic graphs, and larger $k$ can enhance the intervention of the prompt graph in heterophilic graphs.

While these hyperparameters do influence performance, they are not prohibitively sensitive. In practice, prior knowledge of graph homophily can guide the selection of $\tau$ and $k$, reducing the tuning burden. Future work will explore learnable or adaptive $\tau$ and $k$ to automate the process further.

Thank you for your support of our work! We appreciate your insights and have addressed your comments below.

To Weakness 1:

Thank you for pointing out this potential misunderstanding. The notation $\mathcal{G}$ in this figure is indeed unclear.

As we stated in the preliminary section, under the general "pretrain, fine-tune" paradigm of graph representation learning, $\mathcal{G} = (\mathbf{A}, \mathbf{X})$, which means it includes both feature and topological elements. Here, we generalize that input-level prompts can arbitrarily affect both components. We will revise Figure 1 to provide a more detailed and clear representation of this concept in the future version.

To Weakness 2:

Thank you for your valuable suggestions. We completely agree that the introduction to layer-wise prompt methods was insufficient in the initial draft. As you pointed out, while such methods have been less explored in the graph domain, they have already made significant impacts in fields such as vision and multimodal learning. We will include discussions of works from other domains such as VPT [J1] and MaPLe [C1] in future versions, and add detailed analyses between layer-wise prompts in graph domain versus vision/multimodal domains, including design methods, applicable scenarios, and performance differences across different data domains. we will include these discussion in the future version.

To Weakness 3:

Thank you for your suggestions. We provide the fully experiment of Fine-tune and Linear-probe here, as shown in the table below:

From the table, we observe that the performance of Linear-probe is comparable to Fine-tuning across most datasets. Notably, on homophilous datasets (e.g., Cornell, Wisconsin, and Texas), Linear-probe often outperforms Fine-tuning. In contrast, Fine-tuning tends to perform better on some heterophilous datasets.

Comparing this table with Table 1 in the paper, we find that Linear-probe achieves competitive or even superior performance compared to multiple baselines, particularly for representation-level baselines (GPPT and GraphPrompt). The improvement is more pronounced on heterophilous datasets such as Cornell, Wisconsin, and Texas, while there are also slight gains on Chameleon, Actor, and Squirrel. We will include these results in the future version.

To Question:

Thank you for your question. We have discussed part of the potential scenarios in section 5, and we think that overfitting is a common issue in few-shot downstream settings. We attribute this to the use of inappropriate prompt mechanisms, which obscure the prompt's true function, leading to unstable convergence and even negative optimization during training. This phenomenon was part of our motivating experiments.

To address this, first, we designed our prompt using kNN to provide the model with a stable structural prior. This transforms the prompt from discrete, random vectors into a stable structure built upon the semantic relationships between nodes. Second, we employed bootstrapping to further mitigate overfitting, ensuring that both the original topology and the prompt's topology are utilized effectively.

Finally, our approach clarifies the roles of the two key downstream modules: the prompt and the classifier, allowing both to contribute effectively. This not only reduces the risk of overfitting but also ensures that the structural prompt smoothes the optimization landscape, leading to a more stable training process and alleviating potential gradient issues like exploding or vanishing gradients.

Thank you for your constructive feedback; we appreciate your insights and have addressed your comments below.

To Weakness 1:

Thank you for pointing out this potential misunderstanding. We agree $p_{\psi}$ needs clarification. We clarify that, in current practice, $p_{\psi}$ typically consists of two components: a prompt function and a classifier. Many existing works follow this paradigm, focusing on separately optimizing these two modules. However, our analysis in Section 4 aims to demonstrate that the combination of a prompt and a classifier is equivalent to a single classifier. This suggests that such methods may not fully exploit the advantages of prompting. To avoid confusion, we will revise Figure 1 in the future version to more clearly illustrate these two components.

To Weakness 2 & Question 1:

Thank you for your careful review. Indeed, input-level, layer-wise and representaion-level, these three forms represent different scenarios of the three functions. Our formulas are simplified in the Preliminary, which may lead to some confusion. We will provide more detailed formulas here to facilitate better understanding. The optimization objective for all graph prompt learning can be expressed as:

$$\max\_{\Psi} \frac{1}{|\mathcal{D}|} \sum\_{(\mathbf{A}, \mathbf{X}, y) \in \mathcal{D}} \sum\_{i=1}^{N} \log p\left(y\_{i} \mid \text{Predict}\_{\Psi}\left(\mathbf{A}, \mathbf{X}, v_{i}; f\_{\theta}\right)\right)$$

where $\mathcal{D} = \{ ( \mathbf{A}, \mathbf{X}, y )\}$ is downstream dataset, $\Psi$ represents all trainable prompt parameters, $f_{\theta}$ is the frozen pretrained encoder. $\text{Predict}\_{\Psi}(\cdot)$ is a unified prediction function that takes the input graph $(\mathbf{A}, \mathbf{X})$, node $v_i$, the pretrained encoder $f_{\theta}$, to produce the final prediction for node $v_{i}$.

For Input-level, $\Psi$ acts on the input $(\mathbf{A}, \mathbf{X})$, transforming it before it enters $f_{\theta}$. For Layer-wise, $\Psi$ is embedded within $f_{\theta}$'s layers. For Representation-level, $\Psi$ operates on the representations from $f_{\theta}$ (often as part of the classifier), directly influencing the classification.

To Weakness 3 & Question 2:

We appreciate the reviewer gives us the opportunity to clarify. We categorize existing graph prompt methods into three main types: input-level, layer-wise, and representation-level. We categorize these methods based on their mechanisms. We want to analyze these various approaches to gain a deeper understanding of the underlying prompt mechanism.

Regarding representation-level methods, our analysis demonstrates that they are equivalent to a classifier. This suggests that their mechanism contradicts the core advantages of prompting and fails to leverage the benefits of prompts, as similar results can be achieved with a classifier alone. As for layer-wise methods, due to their inherent design complexity and strong reliance on the internal representations of pre-trained models, they are not suitable for black-box pre-trained GNNs. Therefore, we do not consider these methods in our work.

In contrast, input-level methods avoid the limitations and preserve the core advantages of prompting without requiring access to model internals. Thus, they are the most promising among the three categories for our setting. We thus design an input-level prompt mechanism. Through this process, we get our core perspective: "Graph prompt learning should focus on unleashing the capability of pretrained models, and the classifier adapts to downstream scenarios." This viewpoint clearly defines the distinct roles and mechanisms of the two crucial downstream components: the prompt and the classifier.

To Question 3:

We appreciate your thoughtful consideration of our experiments. The Cornell, Texas, and Wisconsin datasets are relatively sparse, with a limited number of edges. When our kNN-generated prompt adds a number of homogeneous edges exceeding the original edge count, the nature of these datasets is substantially changed. This process might even shift the inherent property of data from heterophily towards homophily, in turn, allows the model to perform well on these datasets. Conversely, Chameleon, Actor, and Squirrel are comparatively denser datasets. Adding a specific number of edges to these datasets primarily introduces additional similar information to the nodes. It doesn't really change the dataset's fundamental nature. That's why, even though our model improves performance on these three datasets, the boost is much higher for Cornell, Texas, and Wisconsin.

To Question 4:

We appreciate the reviewer insightful comment. As shown in Figure 4, the performance varies with $\tau$: smaller values tend to perform better on heterophilic graphs, while larger values suit homophilic ones. This aligns with our understanding that adding homophilic bias helps enhance structural learning on heterophilic graphs.

While $\tau$ requires tuning, we view it as part of the prompt design process. In practice, prior knowledge of graph homophily can guide the selection of $\tau$, reducing the search space. Additionally, future work could explore learnable or adaptive $\tau$ mechanisms, which may remove the need for manual tuning altogether. We thank the reviewer for raising this valuable direction.

We thank you for your support of our work. We appreciate your insights and have addressed your comments below.

To Weakness 1:

We have added one graph prompt learning baseline All-in-one [1]. Regarding the 1-shot experiment, as shown in the table below:

We will include these experimental results in the future version.

To Weakness 2:

We thank you for pointing out the shortcomings in our work. We acknowledge that we did not discuss the limitations section thoroughly enough in the initial draft. Here, we add the three types of experiments you mentioned:

Regarding our computational overhead, we provide the 1-shot time and space table for various baselines of DGI-pretrained models, are shown in the table below:

By analyzing the two tables, we observe that our method achieves comparable computational costs compared to various baselines across all datasets. Both the training time per epoch and the GPU memory usage of our approach are reasonable and manageable in practice. These Time and GPU usage tables will be included in future versions.

Regarding the scalability issue, we ran experiments on the large-scale heterophilic dataset arxiv-paper (169,343 nodes, 1,166,243 edges). Here, we use a simplified kNN by randomly sampling 1000 nodes, then connecting each node to its top-k most similar sampled nodes. We test three pretrain strategies under 5-shot setting, comparing with fine-tuning. The accuracy and computational cost are shown in the table below.

As shown in the table, our method incurs minimal preprocessing time and only a slight increase in training time per epoch, with small epoch counts (typically less than 500). This demonstrates that our approach is scalable to large graphs. This table will be included in the future versions.

Regarding the noise robustness issue, we performed Gaussian noise perturbations of different scales on the original features, 1-shot results as shown in the table below:

We observe that at a noise level of 0.01, some datasets (e.g. PubMed, Wisconsin and Actor) maintain their accuracy, which can be attributed to a minor noise-induced augmentation effect. However, a noise level of 0.05 begins to noticeably impact model performance. At a substantial noise level of 0.20, the accuracy across most datasets experiences a significant decline. This analysis will be incorporated into our future version.

From the results in the tables, we conclude that our computational overhead for different datasets is acceptable in practical applications. Moreover, for large datasets, our performance is satisfactory, and the overhead remains within acceptable limits. Regarding the noise issue, from a method perspective, our approach is based on generating prompt topologies from features. Therefore, it encounters problems with noisy features, as this disrupts the completeness of the features. When both features and topology are misaligned with the pre-trained model, our method faces challenges in solving the problem, and our improvement is less ideal. We will include this discussion in the limitations section and present it in future versions. Thank you again for your support of our work.

[1]. Sun, et al. All in One: Multi-Task Prompting for Graph Neural Networks, KDD 2023.