TabGraphs: new benchmark and insights for learning on graphs with tabular features

Thank you for the review and detailed feedback! We answer to the questions below.

Q1. We would like to point out that for different kinds of datasets different types of features (e.g., homogeneous or heterogeneous) are available. In our work, we specifically focus on heterogeneous features not because they are meant to provide better or worse performance, but because for many real-world datasets this is the only kind of features available, and thus designing models specifically for working with such features is important. For this reason, a good benchmark of datasets with such features is required.

Q2. Before feeding into an ML model, strings of text are typically preprocessed using bag-of-words and other text embedding models. These algorithms produce homogeneous features. As we have discussed in the related work section, there is already a vast number of popular datasets with such text-based homogeneous features. In our paper, we specifically focus on graphs with heterogeneous features, which can often be found in real-world tasks, but for which good datasets are currently lacking.

Note that tabular datasets that do not have any strings of text as features are quite widespread in the real world. For example, among the datasets considered in our paper, only questions-tab had strings of text that could be used as additional features. These features were used as the only feature type in the original version of the questions dataset introduced in [1]. However, we removed them in our version of the dataset, as we have found out that, in this particular case, models perform much better with heterogeneous features, which can be seen by comparing the results of GNNs in our paper and [1] on this dataset.

Q3. Indeed, it would be very useful to introduce more datasets from different sciences, but, unfortunately, we do not have access to such datasets. We will continue looking for them, and if you are aware of any quality open scientific datasets with heterogeneous features and additional graph structure, we will be glad to include them in our benchmark.

Q4. In fact, all of the considered models can achieve almost perfect metrics on this dataset with the full feature set (we will clarify this in the paper). This dataset is based on the data from [2]. From the description of the data, we hypothesize that the number of votes was used for constructing the target variable, and thus introduces a data leak. Because of that, we decided to remove this feature.

Q5. The thresholds were chosen so as to produce a graph that is not too sparse and not too dense. Indeed, this threshold can be viewed as a hyper-parameter of graph construction, however, in our paper, we show that the produced graphs are helpful for the task even without searching over the values of this parameter.

Q6. This part of the table is meant to show that such a mix of different types of features that is typical for tabular datasets is indeed present in our datasets. However, even between the features of one type there can be a drastic difference in their importance.

Q7. In the original tolokers dataset from [1], features are processed to make them easy for GNN models to operate on. For our work, we have obtained the original set of features and used it without preprocessing beyond scaling, in line with how it is typically done in tabular machine learning. The features are various performance statistics of workers, such as the number of approved and skipped assignments (numerical features), as well as worker profile information such as their education level (categorical feature).

Q8. We can release the datasets that do not contain proprietary data (tolokers-tab, questions-tab, amazon-users) now. However, for other datasets we need to finalize the legal procedures, which we will do upon the acceptance of the paper.

Q9. Indeed, preparing the graph might take some time. The exact amount varies depending on the nature of the raw data and the type of edges used in the graph being constructed.

For example, if the edges are based on interactions between data samples (e.g., in questions-tab), then constructing the graph is relatively fast, as we simply need to obtain a list of interactions and each interaction corresponds to one edge. On the contrary, if the edges are based on some kind of activity similarity (e.g., in avazu-devices), constructing the edges might take more time as we need to assess how similar two data samples are in terms of their activity.

However, the construction of a graph only needs to be performed once, as part of data preprocessing, so the time required for it is negligible compared to the time spent for running multiple experiments on the dataset. We will add a discussion of this point to the paper.

[1] “A critical look at the evaluation of GNNs under heterophily: are we really making progress?” in ICLR, 2023

[2] “GCN-Based User Representation Learning for Unifying Robust Recommendation and Fraudster Detection” in SIGIR, 2020

Thank you for the review and questions! We address your concerns below.

Although PLR can boost the performance of GNN, it is not novel. It is not surprising that a better node representation can boost the performance of GNN.

Indeed PLR is not novel (and we do not claim it to be). However, it was never used in the graph ML setting before, and our work highlights that it can be an easy way to improve performance on realistic graph datasets with heterogeneous tabular features.

Although the author claimed that these 8 datasets in TabGraph is diverse, their selection criteria is unclear to the reader. In fact, lots of tabular benchmarks contain hundreds of datasets (e.g., AMLB). Benchmarking on only 8 tables may not be conclusive.

For our benchmark, we have specifically selected datasets with available external information about the relationship between data samples that can be used for constructing a graph. Such datasets are much more difficult to find than standard tabular datasets. If you are aware of other realistic tabular datasets with such external information, please let us know so that we could include them in our benchmark.

Note that, in our work, we consider different kinds of relations between data samples — they are based on interactions (e.g., in the questions-tab dataset), activity similarity (e.g., in the avazu-devices dataset), physical connection (e.g., in the city-roads dataset). Importantly, we show that all these relation types can provide useful information to the models that can leverage graph-structured data.

What's the selection criteria of these 8 tabular datasets? For generic tabular datasets, you can build a graph that connects the samples (e.g., based on the sample correlation). Can GNN + PLR still obtain performance in the general setting?

As mentioned above, our aim was to select datasets that have meaningful prediction tasks and contain external relational information, while being quite diverse in terms of tabular feature distribution and underlying graph structure.

Note that directly comparing samples based on their features is not trivial in the case of heterogeneous tabular features (on which we focus in our paper), as these features have widely different scales, distributions, meaning and importance for the task. However, comparing their representation from some layer of a neural network is more meaningful. This is exactly what retrieval-augmented models do (in essence, they simultaneously learn sample representations and a graph based on their similarity). We discuss these models in the related work section and include TabR, a representative example of them, in our experiments. If you know some other meaningful ways to measure the similarity between objects with heterogeneous tabular features for constructing relations between them, please let us know so that we could extend our empirical study.

Thank you for the review and valuable suggestions! We address the questions and concerns below.

The paper should address its focus on either transductive or inductive settings. The comparison of the models deal with both models (gbdt and gnns), and the performance gap could be resulting from the fact that gnn models use the unlabeled test data.

In our paper, we focus on the transductive setting, as it is the most widespread setting in graph ML literature and can be found in many real-world tasks. Indeed, GNNs have an advantage in this setting since they have access to features and incident edges of the test nodes (but not their labels, of course). However, we believe that this advantage is fair, as it is a feature of significantly different design of GNNs and standard tabular models. Also, even standard tabular models like GBDT can potentially leverage unlabeled nodes through pseudo-labeling or other semi-supervised learning techniques. We can extend our experiments with such approaches if required.

In many real-world situations, the data comes with unseen data. The paper should also address the conversion of relations to graphs and application of gnns in those settings.

This corresponds to the inductive graph ML setup. As we have mentioned above, in our paper, we focus on the transductive setting, which is more popular in graph ML literature. However, applying GNNs to nodes unseen during training is not difficult — we just need to add edges incident to the new nodes in the graph and run the GNN on the updated graph once. Extending the benchmark to the inductive setting would be a natural next step in future research.

The paper should address address some of the limitations in converting relations into graphs. For instance, the time it takes to preprocess the data and time comparison of the model evaluations.

Please, refer to our response to Reviewer b5gj in Q9.

The tables of results could be of better quality. (eg, bold-face the best results)

Thanks for this suggestion, we will highlight the best results in our tables.

What are the hyperparameter search space for gradient boosting methods?

The hyperparameter search space for CatBoost:

The hyperparameter search space for XGBoost:

The hyperparameter search space for LightGBM:

Even without the relation (graph-like) information, it is advisable to apply construction of graphs with features (for instance Gaussian-kernel) and then apply one of the comparing methods?

Note that directly comparing samples based on their features is not trivial in the case of heterogeneous tabular features (on which we focus in our paper), as these features have widely different scales, distributions, meaning and importance for the task. However, comparing their representation from some layer of a neural network is more meaningful. This is exactly what retrieval-augmented models do (in essence, they simultaneously learn sample representations and a graph based on their similarity). We discuss these models in the related work section and include TabR, a representative example of them, in our experiments. If you know some other meaningful ways to measure the similarity between objects with heterogeneous tabular features for constructing relations between them, please let us know so that we could extend our empirical study.

Thank you for reading our paper. We reply to the comments below.

with 6-22 features per node

We would like to clarify that our datasets have from 7 to 31 features, please see Table 1.

While new datasets may be useful, the paper does not provide any tangible insights.

In Section 5, we discuss the results of our experiments in detail and highlight the main practical insights with separate bullet points (we also briefly summarize them in the introduction). In particular, we note that one should try to convert relational information (which often exists in tabular datasets) into a graph and experiment on the resulting dataset with graph machine learning methods. In that case, standard graph neural networks augmented with embeddings for numerical features tend to provide the best results, while specialized models previously proposed in the literature do not work very well. We believe that these findings are valuable and useful, especially for those who want to understand which machine learning methods actually work in practice.

First, the datasets are merely described, but there is little rationale about why they have been constructed in this way and what qualitative properties they have.

Our datasets were constructed to cover a wide range of areas and real-world applications where graph-structured data with tabular features appears, and thus to provide a realistic and diverse benchmark. In Section 3.2, we describe in detail various properties of the introduced graph datasets, considering the distribution of node features, various structural properties of graphs, and relations between graph structure and node labels. We also explain why the prediction tasks in our datasets are meaningful and close to real-world applications. If our discussion does not cover some important aspect of the proposed benchmark, and you have any suggestions on how we can improve our analysis, please let us know so that we could include it in the revised version.

Second, and more importantly, the experimental study reports results of some baseline models using fixed (!) hyperparameters, but does not provide any additional insight beyond pure numbers.

This is not correct. As described in Section 5, for the considered GBDT and tabular DL methods, we have conducted an extensive hyperparameter search using Optuna for each of our datasets. Further, we have chosen a fixed set of hyperparameters in EBBS, which the authors claim to work well across different datasets, and conducted hyperparameter search in BGNN using the original experimental pipeline. As for standard GNNs, they take significantly longer to run, so we conducted preliminary experiments in which we determined a set of hyperparameters with which these models provide strong and stable results across a range of datasets, and used it in all our experiments. Note that one of our results is that GNNs typically outperform graph-agnostic models on our benchmark, thus GNNs tend to produce the best results even with less extensive hyperparameter search.

This is more a dataset paper than a research paper.

Please note that we have chosen “datasets and benchmarks” as the primary area of our paper. This topic is also listed in the ICLR call for papers, which shows that our paper fits the scope of the conference. We also would like to point out that empirical ML research is impossible without good datasets, thus dataset papers are of great importance for the research community.

The paper positions itself as graph learning with "tabular" / "heterogenous" features. This raises misleading expectations, as it's really just graphs with a fixed set of vertex-level features.

In our work, we follow the terminology used by previous studies on ML for tabular data in general [1, 2] and graphs with tabular features in particular [3, 4]. In this sense, the proposed graph benchmark represents a collection of tabular datasets with known relations between samples that can be modeled as a graph. If you are aware of other uses of the terms “tabular” or “heterogeneous” in relation to data sample features, please let us know so we could add a discussion of terminology to the paper.

[1] “Revisiting deep learning models for tabular data” in NeurIPS, 2021

[2] “On embeddings for numerical features in tabular deep learning” in NeurIPS, 2022

[3] “Boost then Convolve: Gradient Boosting Meets Graph Neural Networks” in ICLR, 2021

[4] “Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features” in ICLR, 2022