AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization

We would like to express our sincere gratitude to the reviewer for the detailed comments and insightful questions. We respond to each of the reviewer’s comments point by point as follows.

1. "Clarify the hyperparameter tuning process and discuss more on ablation studies."

Thank you for bringing this to our attention. Regarding the hyperparameter tuning process, we pretrain the GFA model with $\lambda,\beta \in \\{1e-1,1e-2,1e-3,1e-4\\}$. Subsequently, we fine-tune these pretrained models and evaluated their performance on the validation set to empirically determine the hyperparameters.

Additionally, we provide a further analysis of the ablation studies below.

(i) The disentangled contrastive graph encoder module is designed to extract discriminative invariant and variant patterns from the data by pulling similar samples closer and pushing dissimilar samples apart in the latent space. Removing this module impairs the extraction of invariant patterns and reduces the distinguishability between patterns extracted from different datasets, ultimately harming the effectiveness of architecture prediction.

(ii) The invariant-guided architecture customization module serves to shield architecture $A$ from the influence of variant patterns $Z_V$ given the invariant pattern $Z_I$. The substantial performance decrease observed upon removing this module highlights the importance of effectively isolating architecture predictions from $Z_V$ influences, reinforcing the critical role of this module in ensuring the invariance conditions of captured patterns.

(iii) This curriculum architecture customization mechanism aims to reduce data dominance in the architecture search process. Removing this module causes certain operations, which perform well on specific datasets during early training stages, to dominate the search process. Consequently, other datasets may neglect potentially beneficial operations.

2. "How does the proposed method compare in terms of computational efficiency with other NAS-based approaches?"

We are grateful for your feedback. In the original manuscript, we analyze the time complexity of GFA as ${O}(|E|d_e +|V|d_e^2 +|\\mathcal{O}|^2d_e +|\\mathcal{O}|(|E|d_a +|V|d_a^2))$. Considering the term with the largest complexity, this is approximately ${O}(|\\mathcal{O}|(|E|d +|V|d^2))$. The time complexity for most existing GNN methods is typically ${O}(|E|d +|V|d^2)$, and GNAS methods also exhibit a approximate complexity of ${O}(|\\mathcal{O}|(|E|d +|V|d^2))$. Thus, our method's complexity is comparable to existing GNAS approaches.

3. "However, the choice of hyperparameters for different datasets is not well discussed, and sensitivity analysis should be enhanced."

Thank you for your suggestion. Since GFA is trained jointly across all datasets, the hyperparameters $\lambda$ and $\beta$ are consistent for all datasets. Specifically, we set $\lambda$ to $1e-3$ and $\beta$ to $1e-1$.

We provide a more detailed discussion on the hyperparameter sensitivity. The hyperparameter $\lambda$ in Eq.17 controls the trade-off between $L_{task}$ and $L_{dis}$. Specifically, $L_{task}$ aims to maximize the mutual information between the invariant pattern $Z_I$ and the architecture $A$, ensuring that $Z_I$ is sufficient to predict $A$. In contrast, $L_{dis}$ aims to minimize the mutual information between the invariant pattern $Z_I$ and the variant pattern $Z_V$, thereby enabling the extraction of two disjoint patterns from the data. We adjust its value within the set $\\{1e-1,1e-2,1e-3,1e-4\\}$. As shown in Figure 5, when $\lambda$ is set too low, the model's performance deteriorates, confirming that proper disentanglement of $Z_I$ and $Z_V$ is essential for effective architecture prediction. Conversely, when $\lambda$ is set too high, performance also declines, indicating that while ensuring the separation between the two patterns, it is equally important that the $Z_I$ retains sufficient information to predict the architecture. Overall, $\lambda$ is an important hyperparameter for balancing the sufficiency and disentanglement. The hyperparameter $\beta$ in Eq.17 controls the trade-off between $L_{task}$ and $L_{inv}$. Specifically, $L_{inv}$ aims to shield architecture $A$ from the influence of $Z_V$ given the invariant pattern $Z_I$. As demonstrated in Figure 5, setting $\beta$ too low results in degraded model performance, underscoring the importance of effectively shielding $A$ from the influence of $Z_V$ given $Z_I$. Thus, $\beta$ is also a critical hyperparameter for balancing the sufficiency and invariance conditions of the patterns captured by the model.

4. "more recent advances in self-supervised learning techniques for GNN adaptation."

We sincerely appreciate your valuable suggestion. In response, we will include a paragraph in the Related Works section of our revised manuscript, discussing recent advances in self-supervised learning techniques for GNN adaptation.

We would like to express our sincere appreciation to the reviewer for providing us with detailed suggestions. We have carefully reviewed each comment and offer the following responses.

1. "Can you explain what claims you want to support with the showcase Figure 4?"

Thank you for highlighting this point. To clearly visualize the customized architectures tailored to different datasets, we presented a heatmap in Figure 4, illustrating the choice weights of each operation at each layer. Firstly, we observe that different graph datasets prefer distinct architectures; for example, Cora mainly prefers GraphConv and GraphSAGE, whereas these two operations are rarely selected for PubMed. This observation further supports our earlier assumption that different datasets require different architectures, and some datasets exhibit inconsistent architectural preferences. Moreover, we find that many datasets prefer varying operations across different layers. For instance, the Arxiv dataset prefers GCN in the first layer and GAT in the second layer. Such fine-grained architectural preferences are challenging to meet through manual design, highlighting the advantage of automated, customized architectures.

2. "How does the computational complexity of GFA compare to classical GNNs and standard GNAS methods?"

We are grateful for your feedback. In the original manuscript, we analyzed the complexity of GFA as ${O}(|E|d_e +|V|d_e^2 +|\\mathcal{O}|^2d_e +|\\mathcal{O}|(|E|d_a +|V|d_a^2))$. Considering the term with the largest complexity, this is approximately ${O}(|\\mathcal{O}|(|E|d +|V|d^2))$. The complexity for most existing GNN methods is typically ${O}(|E|d +|V|d^2)$, and GNAS methods also exhibit a approximate complexity of ${O}(|\\mathcal{O}|(|E|d +|V|d^2))$. Thus, our method's complexity is comparable to existing GNAS approaches.

3. "Could you show the pipeline of GFA duration the inference stage?"

Thank you for highlighting this point. During the inference stage, given an input graph, we first utilize the Disentangled Contrastive Graph Encoder to obtain its invariant pattern representation, denoted as $Z_I$. Then, $Z_I$ is fed into the Invariant Predictor within the Invariant Guided Architecture Customization module to generate a customized architecture. This customized architecture is subsequently employed as the GNN component within the GFM to perform prediction.

4. "Ablation studies seem to validate the effectiveness of the proposed method. But the analyses on the results are a little limited."

Thank you for bringing this to our attention. We provide a further analysis of the ablation studies below.

(i) The disentangled contrastive graph encoder module is designed to extract discriminative invariant and variant patterns from the data by pulling similar samples closer and pushing dissimilar samples apart in the latent space. Removing this module impairs the extraction of invariant patterns and reduces the distinguishability between patterns extracted from different datasets, ultimately harming the effectiveness of architecture prediction.

(ii) The invariant-guided architecture customization module serves to shield architecture $A$ from the influence of variant patterns $Z_V$ given the invariant pattern $Z_I$. The substantial performance decrease observed upon removing this module highlights the importance of effectively isolating architecture predictions from $Z_V$ influences, reinforcing the critical role of this module in ensuring the invariance conditions of captured patterns.

(iii) This curriculum architecture customization mechanism aims to reduce data dominance in the architecture search process. Removing this module causes certain operations, which perform well on specific datasets during early training stages, to dominate the search process. Consequently, other datasets may neglect potentially beneficial operations.

5. "The discussions on the results shown in the figures or tables could be enhanced."

Thank you for your suggestion. We will add more discussions on the results shown in the figures and tables in our revised manuscript to make our paper more readable.

6. "The framework figure 2 is a little simple and did not clearly show the technical details of the proposed method."

Thank you for highlighting this point. We have refined our framework by incorporating additional technical details to better align with the content and equations presented in the Method section, specifically in the following three aspects.

(1) Enhancing the clarity of the pipeline by adding more numerical indexing and clear directional arrows to improve readability.

(2) Including additional equations directly in the figures, enabling readers to easily identify corresponding equations from the text.

(3) Reflecting the unique advantages of our method, such as the customization of different architectures tailored to specific datasets, and the progressive architecture search process driven by curriculum learning.

We sincerely appreciate the insightful comments provided by the reviewer. We have carefully considered each point raised and would like to respond as follows.

1. "The experiment should consider additional way settings beyond the current specific configurations."

Thank you for raising this important point. We have conducted additional experiments to validate our model’s performance under more N-way settings on the Cora and WN18RR datasets. Due to character limits, we only include a subset of the most competitive baselines below. The complete results will be provided in our second-round response and included in the revised manuscript.

Our method outperforms the baselines mostly across various N-way K-shot settings, further verifying the effectiveness of the customized architectures.

2. "additional intuitive explanations could make the theoretical arguments more readable for a broader audience."

Thank you for your suggestion. We provide some additional intuitive explanations below.

Assumption 3.1 assumes that the optimal architectures required by two different datasets may differ. As shown in Figure 1(the performance of each architecture on different datasets), GCN achieves optimal performance on the PubMed dataset, while GraphSAGE performs best on the Wikics dataset.

Assumption 3.1 serves as a prerequisite condition for Proposition 3.2.
Proposition 3.2 demonstrates that when two datasets require different optimal architectures, current mainstream GNAS methods encounter optimization conflicts for GFM. As previously illustrated, the optimal architectures for PubMed and Wikics differ. Consequently, when existing GNAS methods search simultaneously for an architecture optimal for both datasets, they fail to identify a single architecture that performs best for both and are forced to compromise.

Assumption 3.3 defines what constitutes an invariant pattern for architecture prediction.

• Condition 1 indicates that the data contains two types of patterns: an invariant pattern $Z_I$, which reliably predicts the architecture, and a variant pattern $Z_V$, which cannot stably predict the architecture.
• Condition 2 highlights that the variant pattern $Z_V$ are not independent of the architecture $A$.
• Condition 3 states that, given the invariant pattern $Z_I$, the architecture $A$ is independent of the variant pattern $Z_V$, and $Z_I$ is sufficient for predicting $A$.

Proposition 4.1 aims to demonstrate that our method satisfies condition 3 of an invariant pattern for architecture prediction via enforcing P(A \\mid Z_{I}, Z_{V}\) = P(A \\mid Z_{I}).

3. "the paper does not discuss how hyperparameter sensitivity affects performance thoroughly."

Thank you for bringing this to our attention. Due to character limitations, we will provide a more detailed discussion on hyperparameter sensitivity in our revised manuscript. Alternatively, we kindly refer the reviewer to our response to Reviewer YeCs’s Question 3, where we address a similar question. We apologize for any inconvenience this may cause.

4. "The authors should provide more detailed descriptions of the baseline methods."

We appreciate your suggestion. We will include a new section in the appendix providing detailed descriptions of all the baselines.

5. "The framework diagram (Figure 2) should be revised to clarify the technical details of the proposed GFA."

Thank you for highlighting this point. We have refined our framework by incorporating additional technical details to better align with the content and equations presented in the Method section, specifically in the following three aspects.

(1) Enhancing the clarity of the pipeline by adding more numerical indexing and clear directional arrows to improve readability.

(2) Including additional equations directly in the figures, enabling readers to easily identify corresponding equations from the text.

(3) Reflecting the unique advantages of our method, such as the customization of different architectures tailored to specific datasets, and the progressive architecture search process driven by curriculum learning.

We would like to express our sincere gratitude to the reviewer for providing us with detailed comments and insightful questions. We have carefully considered the reviewer's feedback and would like to address each point as follows.

1. "Could you clarify whether the assumptions in the theories can be valid in the real world?"

Thank you for raising this important point. Assumption 3.1 serves as a prerequisite condition for Proposition 3.2. Specifically, Assumption 3.1 assumes that optimal architectures required by two different datasets may differ. To validate this assumption, we evaluated various GNN architectures based on a GNN-based GFM (GFT [1]) across multiple real-world datasets. Figure 1 presents a heatmap visualization of each architecture’s performance on different datasets, showing that optimal architectures indeed vary according to the dataset. For instance, GCN achieves optimal performance on the PubMed, while GraphSAGE performs best on the Wikics.

Assumption 3.3 defines what constitutes an invariant pattern for architecture prediction. The concept of the invariant pattern is well-defined, and methods based on this concept have been validated as effective in various real-world applications, such as academic citation networks [2] and molecular structures [3]. Unlike previous work, which has primarily focused on capturing stable relationships for accurate label prediction, we applied this concept to architecture search, aiming to define invariant patterns that support stable architecture prediction, and designed our method based on this concept.

2. "Could you add more details on the baselines? How do you implement the baselines? The current descriptions are too simple."

Thank you for bringing up this point. For Vanilla GNNs, self-supervised methods, and GFMs, we reproduce the results based on their original papers and publicly available code. To ensure a fair comparison between manually designed GNNs and GNAS baselines, we employ GFT[1] as the base model. Specifically, for manually designed GNNs, we replace the GNN in GFT with various manually designed GNNs and follow identical pretraining and finetuning procedures as GFT. For GNAS methods, we substitute the GNN component in GFT with different GNAS methods. Architecture search is performed during the pretraining stage, whereas in the finetuning stage, we further optimize only the parameters of the searched architectures without additional architecture searches. Furthermore, we use the same search space for both GNAS baselines and GFA, including operations such as GCN, GAT, GraphSAGE, GIN, and GraphConv within a super-network depth of 2 layers. We set the dimensionality of all methods to 768.

3. "More recent advances in GFMs [1] could be relevant references. [1] Graph Foundation Models: Concepts, Opportunities and Challenges. ArXiv:2310.11829".

Thank you for your suggestion. We will add this citation to the related work section of the revised manuscript.

4. "The writing can be improved. For example, more details in Sec. D.2 should be added for reproducing the results. The descriptions on the baselines are too simple, which brings difficulty to the readers that do not very familiar with this topic."

Thank you for your suggestion. We have expanded the description in Sec.D.2 as follows.

We evaluate different GNN architectures and GNAS methods based on GFT[1], following the default hyperparameters of GFT to maintain consistency. To ensure a fair comparison, we set the dimensionality of all methods to 768, use the same search space and operations (GCN, GIN, GAT, GraphSAGE, GraphConv), and fix the number of layers to 2. For our method, we explore hyperparameter $\lambda,\beta \in \\{1e-1,1e-2,1e-3,1e-4\\}$ and empirically set $\lambda$ to $1e-3$ and $\beta$ to $1e-1$. The learning rate of the disentangled contrastive graph encoder is set to $5e-3$, and the learning rate of the architecture predictor is set to $3e-2$. The dimensionality of both the graph encoder and the supernet is 768. Each experiment is conducted 10 times, and we report the average performance along with standard deviations.

We will also include a new section in the appendix with detailed descriptions of all the baselines.

5. "Minors: There are some typos in the manuscript. The corresponding symbols (periods or commas) after equations are missing in Eq. (4) (5) (14) (15) or wrong in Eq. (16). Initial letters should be capitalized in line 434."

We sincerely appreciate your thoughtful observation. We have corrected these typos and will carefully revise the manuscript to address any remaining errors.

[1] GFT: Graph Foundation Model with Transferable Tree Vocabulary

[2] Learning Invariant Representations of Graph Neural Networks via Cluster Generalization

[3] Learning Invariant Molecular Representation in Latent Discrete Space