TabGraphs: A Benchmark and Strong Baselines for Learning on Graphs with Tabular Node Features

Thank you for your review! We answer to your comments below.

On page 2, the authors claimed that GNNs are typically evaluated on homogeneous features, but tabular learning typically has heterogenous features. I think they omit the work of PyTorch Frame (Hu, Weihua, et al. "PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning." arXiv preprint arXiv:2404.00776 (2024). ), which supports encoding heterogenous tabular data through deep encoders and can be used with PyG on downstream GNN tasks. It is good to include comparison to PyTorch Frame + PyG in your paper as it also fits into graph + heterogenous tabular data.

Thank you for pointing out the PyTorch Frame. This work is indeed very relevant and we will include its discussion in our related work section. However, note that PyTorch Frame and our work are complementary: PyTorch Frame provides a library for embedding tabular data with deep models (with the possibility of applying GNNs from PyG to it, among other things), while we provide a benchmark of tabular datasets with additional graph structure, to which the methods from PyTorch Frame and PyG can be applied. The experiments in the PyTorch Frame paper do not consider such datasets, they only cover standard tabular datasets with no additional graph structure and RDB datasets from RelBench (see our general response for a detailed explanation of the differences between our datasets and RDB datasets). We are glad that the intersection of tabular and graph machine learning is starting to attract more attention.

The authors claim on page 4. "In contrast, our work is focused on single-table data with a single type of relationships between entities in the same table.". However, it seems to me all the graph structure constructed by the authors can be represented by the graph structure in Relbench as well.

Please see our general response for a detailed explanation of the differences between our single-table data setting and the setting of ML for relational databases (RDBs) which is considered in RelBench. While single-table data can be considered as a special case of multi-table RDB data, this is a very important special case with its own specifics and lots of practical applications, and it potentially requires different methods from those used in the general case of ML for RDBs.

I also think the task type is too limited. Currently it only supports node classification and node regression. The authors did not mention link prediction tasks. Link prediction is a common tasks solved by GNNs and is offered by Relbench. It is unfair to say node level task is "the most common setting in modern graph machine learning".

Link prediction is an important task. However, it provides a lot of degrees of freedom such as negative edge selection for both training and evaluation, which makes designing a benchmark for link prediction a very complicated task. There are works dedicated specifically to this purpose [1]. Thus, we believe that the link prediction task is out of the scope of our work. We would also like to note that most GNN papers only evaluate their models on node-level tasks, so we believe that node level tasks are indeed the most common setting in modern graph ML.

[1] Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking (NeurIPS 2023)

In the experiment section, the authors only compared the performance scores. I think it's also good to compare the runtime of the methods, as GBDTs can be much slower on larger datasets as compared to GNNs and tabular models.

We provide the runtimes of the considered models in Appendix F. Note that graph-agnostic models are typically faster than graph-aware ones since they do not need to process the graph structure (such processing is not very hardware-friendly and thus often takes much time). In particular, due to LightGBM being highly optimized for performance, it is actually the fastest of the methods considered.

The authors propose two types of tasks, node regression and classification. I'd like to see some other task types being covered, for example, multi-label classification, or link prediction. For example, in the hnm dataset, can you add a task to predict new items that will be co-purchases?

Predicting item co-purchasing is indeed a practically important task. However, we believe it would be more suitable for a work centered on recommender systems. Also see our response about the challenges of designing benchmarks for the link prediction task above.

The authors should make proper comparisons to existing relational learning benchmarks RelBench and 4DBInfer.

Please see our general response.

Thank you for your review! We address your concerns below.

Lack of Novelty. The concept of constructing graph datasets from tabular data is not novel and has been previously suggested in multiple studies.

While the concept of constructing graph datasets from tabular data is not novel (and we discuss previous works on this topic in our paper), there is a lack of open datasets that represent this type of data (and this lack was acknowledged in previous works), which impedes research on this topic. Thus, we believe constructing a benchmark of such datasets is very important.

Writing. The initial paragraph in the 'Machine Learning for Graphs' section of the related works is excessively lengthy, potentially hindering readability. Besides, The main text should provide more details about how tabular data is represented as graph data. The absence of these details in the main text may lead to confusion among readers.

We will restructure our related work section to improve its readability. As for how graphs are constructed for tabular data, we specify what relationships are represented by graph edges for each dataset in the main text, and provide details about the graph construction in the Appendix (since these details take up a lot of space, and moving them to the main text will seriously reduce readability and leave almost no space for other parts of our work).

More discussion is needed on the differences between this paper and the previous paper on this topic.

We discuss the difference between our work and previous works on learning on homogeneous graphs with tabular node features in the “Machine learning for graphs with tabular node features” paragraph of the related work section. In short, previous works focused on building complicated models, while acknowledging the lack of open datasets to evaluate these models. Our work provides such datasets, and also proposes simple modifications to existing tabular and graph ML methods that outperform complex models proposed in previous works. We also discuss the differences between our work and works on ML for relational databases (RDBs) in the last paragraph of the related work section and in Appendix C. In short, ML for RDBs deals with data obtained from multi-table RDBs and represents it as heterogeneous graphs, while our work deals with data obtained from single tables and represents it as homogeneous graphs (see our general response for more details).

Why there are models that suffer from performance drop after utilizing PLR?

There can be many reasons for why PLR does not improve performance across all models and datasets. For instance, PLR introduces additional hyperparameters and trainable parameters — the former can be difficult to choose optimally, while the latter can lead to overfitting.

Thank you very much for your response. I have read it carefully and will maintain my score.

Thank you for your review! We address your questions and concerns below.

W1. Please see our general response for a detailed discussion of the differences between our setting and ML for relational databases (RDBs). Indeed, our single-table data setting can be viewed as a special case of RDBs with one table and additional relational information. However, this is a very important special case with its own specifics and many practical applications, and it potentially requires different methods from those used in the general case of ML for RDBs. In the same way, textual data can be viewed as a special case of data consisting of both texts and images, and appropriate models (vision-language models) and benchmarks for the latter already exist, but it does not mean that the models and benchmarks for purely textual data no longer need to be developed. Both ML for RDBs and our work aim to bring graph ML methods to tabular data, but our benchmark and RDB benchmarks use different sources of data, different ways to represent it, and different methods of working with it. We believe that both directions are important and deserve attention from the community.

W2. Our aim was definitely not to “hide” this discussion. In fact, our discussion of ML for RDBs is rather extensive, which is why most of it was moved to the Appendix. However, we provide a short summary and a reference to the Appendix in the main text. Following your suggestion, we will try to improve the structure of our paper by providing more details in the main text. Further, everything in our discussion of ML for RDBs can be applied to the RelBench and 4DBInfer works (among others mentioned in our paper), thus it is misleading to say that there is only 1 sentence about them.

W3. The aim of our work is to bridge the gap between tabular and graph ML, thus we believe the discussion of these two fields is quite relevant. Nevertheless, we will try to improve the structure of our paper. Also note that already the second paragraph of the introduction discusses examples of realistic scenarios where a graph structure naturally arises in tabular data, which is exactly the focus of our work.

W4. The ICLR 2025 call for papers lists “datasets and benchmarks” as one of the relevant topics for the conference, and we have appropriately specified “datasets and benchmarks” as the primary area of our work. Thus, we believe our work fits the scope of the conference. We also would like to point out that empirical ML research is impossible without good datasets and benchmarks, so they are of great importance for the research community. Further, one of our contributions is providing simple modifications that help improve performance of existing models and often achieve the strongest results on the newly proposed datasets.

Q1. Please see our comments above and our general response about the differences between our setting and ML for RDBs. We extensively discuss ML for graphs in the related work section because most modern works about graph ML consider homogeneous graphs, which is exactly our setting, so these works are directly relevant to ours. In contrast, ML for RDBs works consider a different setting with heterogeneous graphs obtained from RDB data, thus being less relevant to our work (but still relevant, of course, so we discuss them briefly in the related work section and in much more detail in the Appendix).

Q2. Working with homogeneous graphs is neither a benefit nor a limitation. Some types of data can be represented as homogeneous graphs, and other types of data can be represented as heterogeneous graphs. Appropriate methods need to be used for both types of data, and benchmarks providing such types of data are needed for the development of these methods. There are many GNNs for heterogeneous graphs that are extremely different from GNNs for homogeneous graphs (for example, some of such GNNs are discussed in [1]). Once such models are developed, swapping one model for another indeed takes a few lines of code, but that does not mean that the development and evaluation of these models is not significant. Using a heterogeneous GNN on a homogeneous graph will not provide any additional expressivity, but will only make the model unnecessarily more complex. Further, there is a lot of current research on GNNs for homogeneous graphs (in fact, much more than on GNNs for heterogeneous graphs), which suggests that the community mostly does not consider that GNNs for homogeneous graphs do not have any benefits. While there are of course advantages to trying to cover the most general case possible, such an approach can also unnecessarily complicate the development and application of ML methods. Thus, it is important to be careful about using appropriate methods for different types of data.

[1] Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks (KDD 2021)

Thank you for your reply. Please note that we never claimed that tabular data cannot be viewed as a special case of RDBs (and we are not aware of any researchers claiming that, which is not surprising, because such a claim is incorrect). What we claim is that most researchers in the field of ML for tabular data do not consider viewing general tabular data as a special case of RDBs as a useful approach, as evidenced by the vast majority of research on tabular data not mentioning RDBs at all. As a few examples, [1-3] are recent papers conducting evaluation of a large number of tabular ML models on a large number of diverse single-table datasets. Together, they use more than a hundred tabular datasets. RDBs are not mentioned even a single time in these works. If you are aware of research that studies the single-table data setting and discusses it in the context of RDBs, we would be grateful if you could share it with us, because we are not familiar with any such works.

We would like to emphasize that we do not aim to downplay the importance of ML for RDBs — huge amounts of data are stored in RDBs, and applying appropriate ML methods to them can lead to great benefits. But there are also vast areas of tabular ML research and practice that do not view themselves as a part of the ML for RDBs field, and we aim to show that graph ML methods can be very useful there too.

[1] Why do tree-based models still outperform deep learning on tabular data? (NeurIPS 2022)

[2] When Do Neural Nets Outperform Boosted Trees on Tabular Data? (NeurIPS 2023)

[3] TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (preprint 2024)

Thank you for your review! We address your concerns below.

1. Please note that this survey does not explicitly discuss any existing datasets. We have checked which datasets are used by other works discussed in the relevant sections of this survey, and these datasets are either not open, or are among the datasets that that have various flaws, which we discuss in Appendix F, or do not have explicit graph structure at all (it is suggested to construct graph edges based on feature similarity or to learn graph edges from data in an end-to-end way, which are not the approaches of our work). Thus, we believe that our benchmark is an important contribution to the field. We provide open realistic datasets with explicit graph structure which is based on additional external information not contained in node features.

2. Our experience suggests that there can be no “standardized mechanism or methodology to augment any table with graphs”. Data from different domains requires different ways of constructing a meaningful graph, and for some types of data there is probably no way to construct a meaningful graph at all. However, there are many important real-world applications where a graph can be constructed very naturally — social networks, traffic networks, internet, co-purchasing networks, and many other applications, which we discuss in the second paragraph of the introduction. People working with a particular type of data can often very easily come up with a way to construct a meaningful graph from this data. Our datasets provide a wide range of realistic examples of this.

3. Note that the phrase “it is important to experiment with different design choices” in our work refers specifically to standard GNN architectures, and we have shown that some other more advanced and possibly over-engineered methods do not work as well, thus pruning the search space. Moreover, we believe that fair evaluation of different competing models is a useful result. If there is no one-size-fits-all solution (and our experiments show that this is the case), it is important for researchers and practitioners to know about it so that they can conduct the necessary model selection to obtain better results for their specific data, instead of believing that there is a single model that universally achieves the best results in all situations (which is how many models are presented nowadays — quite often misleadingly).

4. While techniques similar to NFA have been used in prior works (as we discuss in our paper), our aim was to highlight the usefulness of this method for learning on graphs with tabular node features. This method was previously overlooked in this setting (we are not aware of any prior works that apply NFA-like techniques to tabular node features), whereas it outperforms much more complicated models that have been proposed previously. We believe that showing this is a useful additional contribution of our work. Also note that the paper on PNA that you cite does not experiment with any datasets with tabular node features, and their method of combining multiple neighborhood aggregation is used as part of GNNs, while we propose using multiple neighborhood aggregations as a feature preprocessing step for graph-agnostic models, which is much simpler and allows for using much more computationally efficient models.