Towards Neural Scaling Laws for Foundation Models on Temporal Graphs

W1. Contribution of the paper

Thank you for raising this point. In this work, we focus on demonstrating the potential of foundation models for temporal graphs. We believe our contributions are significant in several ways:

1. Multi-Network Training Algorithm: While most methods in temporal graph literature are trained and evaluated on a single network, we propose the TGS algorithm, which enables effective multi-network training.

2. Transferability Across Networks: We are the first to show strong transferability across networks, which is a key characteristic of a foundation model and crucial for its generalization.

3. Performance Improvement: Our MN model outperforms single-network models on unseen networks, demonstrating the advantages of multi-network training and the effectiveness of foundation models in temporal graph learning.

By introducing the first multi-network temporal learning approach on a real-world dataset and demonstrating performance improvements over single-model approaches, we aim to open a new research direction toward temporal graph foundation models. This work represents a significant first step in advancing the understanding and application of foundation models in temporal graph learning, and we believe it will contribute substantially to the ongoing development of this emerging field.

W2. More baseline methods

We appreciate the reviewer’s feedback regarding more baseline models. It is worth mentioning that our work primarily focuses on graph-level tasks in the Discrete Time Dynamic Graph (DTDG) setting, while models presented in TGB leaderboards and DyGLib are designed for the Continuous Time Dynamic Graph (CTDG) setting for node and link level tasks. Our work also includes GraphPulse [1], the state-of-the-art model for graph property prediction in DTDG, and three other common methods for DTDG. To further address your concern, we evaluated another baseline T-GCN [2], as shown in the table below, to make the comparison with more baselines. Without needing any additional training on unseen networks, our MN-64 model outperforms the single-network T-GCN in 18 out of 20 unseen test networks, demonstrating significant transfer capability and highlighting the advantages of our multi-network training approach.

Comparison of new baseline’s, T-GCN, performance and MN-64

[1] Shamsi, Kiarash, et al. "GraphPulse: Topological representations for temporal graph property prediction." The Twelfth International Conference on Learning Representations. 2024.

[2] Zhao, Ling, et al. "T-GCN: A temporal graph convolutional network for traffic prediction." IEEE transactions on intelligent transportation systems 21.9 (2019): 3848-3858.

Thank you for your valuable feedback which helps us improve the clarity of the paper. We wanted to take this opportunity to clarify our work further. In our study, we trained a temporal model on 64 distinct token networks, each representing a separate financial network with its own set of investors, varying durations, and unique investment behaviors. This is the first study to train a temporal model on such disjoint networks, where each network is independent and exhibits different investment dynamics., With our proposed multi-network training algorithm TGS train, our model is able to effectively capture common structural evolution across these diverse financial networks, demonstrating its ability to generalize and identify shared behaviors.

To evaluate the performance and generalization of our model, we tested it on 20 unseen networks, which share only 2% of their nodes with the training set. This test setup is particularly challenging, as prior works like TGAT, CAWN, and NAT [1,2,3] typically rely on a test set with 10% unseen nodes for inductive testing while our test networks have 98% inductive nodes. The minimal node overlap further underscores the robustness of our model, as it demonstrates the ability to transfer learned knowledge to networks with new investors and behaviors with no fine tuning. By presenting this work, we aim to open a new research direction in temporal graph learning, specifically in multi-network training. Additionally, we will make our datasets available to the research community to encourage further exploration in this area.

[1] Xu, D., Ruan, C., Korpeoglu, E., Kumar, S., & Achan, K. (2020). Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07962.

[2] Wang, Y., Chang, Y. Y., Liu, Y., Leskovec, J., & Li, P. (2021). Inductive representation learning in temporal networks via causal anonymous walks. arXiv preprint arXiv:2101.05974.

[3] Luo, Y., & Li, P. (2022, December). Neighborhood-aware scalable temporal network representation learning. In Learning on Graphs Conference (pp. 1-1). PMLR.

I really appreciate authors' efforts in explaining details as well as the additional experiments carried out to address my concerns. I checked carefully all of the changes and responses from the authors. Now I am very sure about my judgment and I still hold my view towards a rejection. Sorry for that.

1. As I mentioned, in my opinion, scaling law should not be proven by only considering the increasing amount of training data. I think studying the model size and architecture is also important. In NLP, major SOTA models are Transformer-based and this architecture is empirically proven to be effective. So people are trying to stack more Transformer layers and leveraging more training data which uncovers the scaling law. In this submission, I can only find that the authors are trying to relate to scaling law by introducing more training data on a fixed model structure taken from a baseline model. The choice of the architecture of the baseline model is not perfectly motivated since dynamic graph representation learning methods are diverse in their architectures, not like the role of Transformer in language models.

2. Another point is that, dynamic graphs can be continuous (CTDGs) or discrete (DTDGs). If we want to show the scaling law, it is not very comprehensive to only focus on DTDGs. I think this is also a great concern.

3. As pointed out by Reviewer zwwh, I also agree that the transferability across different domains should be considered. It is a very important characteristic when we refer to scaling law. In NLP, language foundation models are able to do zero-shot inference in the domains that are not frequently observed in the pre-training data (lets call these domains sparse domains here). Of course we cannot expect a strong performance in these sparse domains, but their performance can be improved by scaling the model size as well as the pre-training data. So I believe that it is important to consider cross-domain transferability when we discuss scaling law.

To summarize. I am certain that this work has merits and it is novel, but I think bringing up the notions of "neural scaling laws" and "foundation models" is overexaggerated. Personally, I think the experimental settings and the results cannot lead to constructive conclusions related to neural scaling law on dyanamic graph representation learning. The initial version of the paper was not very consistent in form. After the revision, I can see improvements, but I still believe the storyline of this paper has flaws. So I will keep my score unchanged.

W1. Novelty of the paper

Thanks for your comment. Our main contribution is to demonstrate the first successful transfer of tasks across temporal graphs: our multi network model achieves state-of-the-art performance on unseen test networks without any fine tuning, outperforming models that were trained specifically on these test networks. Our work enables the study of scaling laws and the application of foundation models across diverse temporal graphs, opening up new possibilities for generalized learning across different networks. This highlights the powerful transferability and scalability of our model and in learning the evolution of temporal graphs and applying this knowledge to new, unseen networks. In this sense, our work presents an important advancement in the field of temporal graph learning by introducing the first multi-network model for temporal graph learning. Unlike existing approaches, which focus primarily on static graphs or training on single temporal networks, we propose a novel multi-network training scheme. Additionally, our work provides the largest collection of temporal graphs, accompanied by tools that enable the construction of foundation models for temporal graph learning. We believe these contributions are highly significant for ICLR as they not only pave the way for future research on temporal graph foundation models but also offer a valuable resource to the research community.

W2. Clarification of graph property prediction and definition of temporal global efficiency, temporal-correlation coefficient, and temporal betweenness centrality

Thank you for your insightful feedback. In this work, we focus on temporal graph property prediction which aims at forecasting properties about the temporal graph for example the growth or contraction of transaction volumes in future weekly snapshots, utilizing the number of transactions from the prior week. Our objective is to determine if a network will exhibit growth or shrinkage in transaction activity during the upcoming week based on the current weekly snapshot. This analysis is particularly crucial in financial networks, as it aids in predicting investor engagement and potential fluctuations in transaction volume, subsequently influencing investment strategies.

In response to your constructive comment, we have added Section C in the appendix, which presents a formal definition of our property prediction framework along with supplementary details. The property prediction established in this study pertains to the growth or shrinkage of transaction volumes, specifically focusing on the alteration in the number of transactions from one week to the next. The formal definition of graph property prediction is articulated as follows:

Graph Property Prediction Definition

Let G represent a graph, t a specific time, and E(t₁, tₙ) the multi-set of edges between times t₁ and tₙ. The property P is defined as the change in edge count (number of transactions) between consecutive weekly snapshots, as shown:

P(G, t₁, tₙ, δ₁, δ₂) = 1, if |E(tₙ + δ₁, tₙ + δ₂)| > |E(t₁, tₙ)| 0, otherwise.

In this study, we set n = 7, δ₁ = 1, and δ₂ = 7, which implies that we are predicting transaction volume changes over a 7-day span. This forecasting is especially pertinent in the financial sector, where understanding transaction growth or contraction can inform investment choices and improve comprehension of market dynamics. Additionally, we identify several other potential property predictions within temporal graphs that warrant exploration in future research. These include temporal global efficiency, temporal-correlation coefficient, and temporal betweenness centrality, among others.

In Section C of the appendix, we delineate formal definitions and discuss the applicability of each to financial networks, illustrating their relevance to transaction networks and the insights they may provide for financial applications. By investigating these property predictions, we aim to enhance the understanding of temporal graph analysis and motivate further investigation by the research community utilizing our datasets.

W1.General transferability on different domain

Thank you for your valuable comment. To the best of our knowledge, this is the first study to demonstrate the transferability across temporal networks within the same domain, establishing a promising direction for future research. To better clarify the different structures of our datasets we provide Figure which highlights the substantial diversity in graph structures and distributions in our datasets. We also compared our dataset diversity with the available dataset in [1] which further illustrates how our proposed TGS datasets improved upon the available ones. While we acknowledge the importance of generalizing across multiple domains, we believe the first step is to prove that such transferability is possible within a single domain and provide datasets for the research community to explore this important topic. We agree that learning from multiple domains is an interesting next step which we leave as future work.

[1] Shamsi, K., Poursafaei, F., Huang, S., Ngo, B.T.G., Coskunuzer, B. and Akcora, C.G., 2024, March. GraphPulse: Topological representations for temporal graph property prediction. In The Twelfth International Conference on Learning Representations.

W2. The training method is based on HTGN

This is different from most TGL work where a single network is trained and tested on, testing completely on unseen network is new and the setting of multi-network training is also new.

Generalization to unseen temporal networks, we assume all networks come from same IID distribution, same distribution for train and test.

We appreciate your valuable feedback. To address the concern about the generalizability and robustness of our results across various GNN architectures, we have included an additional model in our evaluation: GC-LSTM. This model combines graph convolutional networks with long short-term memory units, enabling it to effectively capture both spatial and temporal dependencies in graph data. It serves as an important benchmark for testing the temporal robustness of our findings. Both the GC-LSTM and our HTGN-based model were evaluated under identical conditions, and the results are now presented in the revised manuscript and Figure of the rebuttal PDF. While GC-LSTM demonstrates some scaling behavior, our HTGN-based model outperforms it, exhibiting superior scalability across networks.

W1. Ablation study with larger MN

We thank you for this suggestion. We have added ablation study on MN-16, MN-32 and MN-64. We observe that the conclusion remains the same: both IID training and Context Switching are essential parts of the TGS training algorithm and each provides a performance boost when applied.

W2. Evaluation on other prediction tasks

Thank you for raising this point. To the best of our knowledge, we are the first to investigate neural scaling law on temporal graphs. Therefore, we would like to focus on a task that is defined over the overall graph structure and potentially transferable across graphs thus choosing the graph property prediction task.

A significant portion of new nodes is introduced at each snapshot. In the following table, we compare the number of new nodes coming into the network between pairs of consecutive snapshots, represented as (t, t+1), (t+1, t+2), and so on. The first column shows the average percentage of new nodes introduced between each snapshot pair, while the second column reports the highest percentage of new nodes observed between any two consecutive snapshots. On average, we observe that approximately 20% of nodes are new with each snapshot, demonstrating the dynamic and evolving nature of the network. Some datasets show a much higher percentage of new nodes, even exceeding 90%, which is common in financial networks as investors frequently enter and exit the network. This high turnover of nodes reflects the changing nature of participants in financial systems, where new investors may join and others may leave. This high turnover of nodes makes it challenging for models to effectively learn node and link-level tasks, as these tasks rely on a relatively consistent set of nodes across snapshots.

However, graph-level tasks are more well-defined and offer significant advantages in financial network applications. In particular, predicting graph properties can provide deeper insights into the structural evolution of networks, which is crucial for identifying investment signals in practice. Given this, our work focuses on graph-level tasks to leverage the dynamic nature of financial networks and deliver actionable insights for real-world applications.

2. Node Overlap Analysis

Based on the comments of the reviewers, we have conducted a computation to highlight the overlap of new nodes in our training and test sets showing that there is minimal overlap between training and test nodes.

We analyze the overlap of nodes between different datasets and within each dataset, which helps demonstrate the highly dynamic nature of our datasets. Specifically, we compared the nodes in each test network with those in the training networks and calculated the average overlap. The results showed that, on average, only 2% of the nodes are common between the training and test networks, highlighting the rapidly changing structure of these networks. Furthermore, we analyzed the node overlap within each test dataset by splitting it into the standard train-validation-test setup. We compared the nodes in the 70% training snapshots with the nodes in the final 15% test snapshots, and on average, only 4% of the nodes overlapped. This indicates the highly inductive nature of our model and emphasizes the zero-shot challenge it addresses in this domain. These findings underscore the importance of tackling such dynamic and evolving challenges in temporal graph learning. We have also included this information into the appendix section G of the paper

Table 1: Overlapping Nodes Statistics