Q: What is the definition of data leakage, and why is it a problem? Similar distribution with training and testing is not necessarily called data leakage.

Regarding data leakage - we use this term for datasets that include information that's unavailable during model deployment. This is a common definition (Chip Huyen, chapter on feature engineering, Kaggle ML Course, Sayash Kapoor et al.). We will add a note on this to the paragraph about leaks in section 3 shortly. In some datasets, one of original target variables is added to the features (e.g. SGEMM_GPU_kernel_performance, sulfur), while some datasets have an answer to one example as a feature in other, and originally used non-random split to avoid leakage, which was changed to random in later literature (eg. Facebook Comments Volume, electricity).

One of the main arguments is data leakage; however, only 11 (10%) were detected among 100 datasets, which is not significant.

As we discuss above, pointing out data-leakage issues is not the main argument in our paper. Nevertheless, it is an important observation we make from dataset analysis. We do not agree that this is not significant, 11% can noticeably affect the evaluation results.

Experiments and Technical Details

While there were some models inside, the comparison was made only to one single framework. The related work mentioned more than one. Some experiments may be helped to pursue reader the importance of TabReD.

We believe that the breadth of comparison in Figure 1 is not a priority, as it is addressed by:

1. Consistency of our results obtained on the Gorishniy benchmark with the results from the tabular DL literature. For example, retrieval methods (TabR, ModernNCA), embeddings and ensembles show consistent performance across hundreds of datasets in existing benchmarks (the largest one to date is Han-Jia Ye et al., it covers many datasets from Léo Grinsztajn et al. and Tabzilla). Strenghts of the long-training with augmentations is also replicated on a broader set of datasets (Kyungeun Lee et al.).
2. Extensive testing on TabReD under controlled conditions:
   • Analysis of temporal vs random splits impact on model ranking in Section 5.4
   • Analysis of the feature-rich vs "typical academic" feature-engineering regimes on model ranking in the rebuttal experiments (see general responsе), we'll update the pdf with these additional results shortly.

Experimental results demonstrate that both temporal drift and extensive feature engineering present in industrial data lead to qualitatively different model behaviors compared to existing benchmarks.

The paper has many details, but they are written within the appendix. The main pages lack technical details. To understand the paper completely, the 10 main pages are not sufficient. This is actually a very long paper.

We believe that the size (9.5 pages) of the paper is justified by many figures and detailed discussion of the new benchmark position and motivation. What specific missing details are you referring to?

Thank you for the review. We address your concerns and reiterate relevant paper contribution below.

Clarifying the core contribution of the paper

Distinction between "Framework" and "Benchmark":

...the author introduces a new framework called TabReD...

TabRed, we do not need to reinvent the wheel for each problem. Gorishniy's framework is more than enough for the experiment. If your focus is on the above problems, then you can just adjust the Dataloader class from Pytorch

It is important to clarify that our main contribution is in introducing new datasets from real-world tabular problems with temporally shifting data and extensive feature-engineering that are often encountered in industry but rarely available in existing benchmarks. We do not introduce new frameworks in the submission and use an established protocol/framework (by Gorishniy et. al.) to evaluate popular techniques.

We also actively discourage viewing this benchmark as a replacement for existing tabular benchmarks in section 4.1. We provide this benchmark for testing methods in previously underrepresented conditions and do not discourage testing on random splits and previously available datasets.

Core contribution - the need for representative datasets

The main issues are domain shifting and time shifting. Both areas are not new and have been explored very well. Domain shifting is also called an out-of-domain or out-of-distribution Problem. This problem is not new and is well-known within academic research.

We discuss relevant prior work on temporal shift and domain shift in both general ML (Yao, Huaxiu, et al.), tabular DL (Sergey Kolesnikov et al., Josh Gardner et al.) and specific applications (Roger M. Stein, Chip Huyen, Daniel Herman et al.) in the related work section. One of the key insights from this line of work is precisely that evaluating methods on identically distributed data may not predict real-world performance when temporal shift is present, which brings us to the second point

For the second problem, time shifting split is a vague problem. There are reasons why random split is widely used. The main reason is that a random split is expected to have a similar distribution, unlike a shifting split. The result of a model trained in a time-shifting split is expected to vary between experiments; this is just because it came from a different distribution.

We cannot agree with the reviewer's claim. First, temporal Shift is a concrete and common phenomenon, even amongst existing datasets (see numerous sources we cite above, in related work and in section 3 discussion). Random splits may not be indicative of real-world performance in deployment scenarios where data naturally evolves over time. As we demonstrate experimentally in section 5.4 methods' relative performance can change significantly when evaluated on time-based splits that better match deployment conditions. This is a systematic difference, not merely higher variance due to a different distribution.

Furthermore, the temporal shift is not the sole characteristic of the new datasets. In addition to temporal splits, TabReD datasets have rich feature sets (median 261 features vs 13-23 in existing benchmarks, see [Table 1]). This reflects substantial industrial investment in feature engineering, as detailed in Section 4. This characteristic of TabReD datasets also affects existing tabular DL techniques differently (see the new results on this in the general response).

Both temporal-shifts and extensive feature engineering are characteristics common in industrial applications that are underrepresented in the existing benchmarks. We cover this specific scenario with TabReD. As we state in section 4.1, TabReD serves as a further evaluation for demonstrating industrial usefulness of novel techniques in tabular DL. We do not discards existing IID benchmarks, which are still important and useful.

Thank you to the authors for their hard work; I truly appreciate the update.

Data Leakage is a common term, but it has several meanings. For example, in security, Data leakage could mean a data breach. [1]. The paper mentioned was written in 2016.

As we discussed above, pointing out data-leakage issues is not the main argument in our paper. Nevertheless, it is an important observation we make from dataset analysis. We do not agree that this is not significant, 11% can noticeably affect the evaluation results. I agree that this may not be the primary focus of the paper. Still, I believe it serves as an essential foundation for justifying it, especially as it challenges other available benchmarks that use the 100 datasets.

We believe that the size (9.5 pages) of the paper is justified by many figures and detailed discussion of the new benchmark position and motivation. What specific missing details are you referring to?

In my understanding, a key objective of this paper, as highlighted in the abstract, is: We demonstrate that evaluation on time-based data splits leads to different methods ranking, compared to evaluation on random splits, which are common in academic benchmarks. F. However, we could not find specific technical details on how this is implemented in Section 5.1. Instead, it is suggested to refer to the appendix for further information.

For extended description of our experimentalsetup including data preprocessing, dataset statistics, statistical testing procedures and exact tuninghyperparameter spaces, see Appendix C. Below, we describe the techniques we evaluate on TabReD.

I am not addressing the concept of shifting distribution, as it is indeed a valid phenomenon. Instead, I am focusing on the time split method utilized by the authors. The authors did not demonstrate that implementing a time shift split leads to a distribution shift. Rather, they presented performance variations, which are anticipated when a non-random shuffle is applied. I believe time shifting leads to temporally static, which tends to overfit [2]. And I believe that is the reason for the big variation in performance.

[1] A survey on data leakage prevention systems. 2016 [2] Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation

The authors did not demonstrate that implementing a time shift split leads to a distribution shift.

The table above demonstrates the change in relative performance between methods due to timeshift and richer features. While we believe the change in relative performance gives motivation to use temporal splits, which are closer to practice for applications described in this paper, we would like to closely focus on why we believe they are explained by distribution shift.

First, the explanation from your [2] citation does not contradict the existence of distribution shift in time. The differences in some features that arise when applying time split could be explained by temporal distributional shift. Second, "Wilds: A Benchmark of in-the-Wild Distribution Shifts", (by Pang Wei Koh and Shiori Sagawa et al.), define distribution shifts as a situation where "the training distribution differs from the test distribution". We would like to highlight that on our datasets, not only the distribution of some features changes from train to test, but these features are important for the models we used. We measure Wasserstein distance between train and test distributions of the target variable Y for the random and temporal data splits and an average Wasserstein distance between train and test for top-20 most important features. The distances are summarized in the following table:

For all datasets both target and top features distributions are further apart for the temporal data splits, indicating the presence of shift. Similar notions of distribution shifts were used in Gardner et al. for indicating target shifts for example (they used a simpler, absolute difference between average targets for train and test sets)

However, we could not find specific technical details on how this is implemented in Section 5.1. Instead, it is suggested to refer to the appendix for further information.

We changed the experimental setting to a comparison between canonical TabReD and the same data but split randomly on train/val/test parts for simplicity and included these details in the section (instead of using different sliding window splits in this section). We also updated section 5.4 from only focusing on the influence of time shift to also include the experiment on the influence of richer feature sets.

To sum up this response:

1. We actively avoid challenging previous benchmarks, and think they do a great job with demonstrating behaviour on non-timeshifting and non-feature rich data.
2. We demonstrate that method ranking changes when we "turn off" the two novel conditions described in this paper. This demonstrates that random-split evaluation could lead to wrong conclusions for applications on our data, since they perform on data in the future relative to the one on which they were trained.
3. Using the definition of distribution shift from "Wilds: A Benchmark of in-the-Wild Distribution Shifts", we demonstrated that in our datasets, there is a distribution shift between train and test subsets.
4. We changed the paper to be more intuitively understandable, study the effects of both of our new settings instead of just one, and create less impression of deprecating previous benchmarks.

Since the discussion period is almost over, could you reiterate the concerns about our paper that you still have after this response? We would greatly appreciate it and it would help us improve our work.

The role of TabReD benchmark

According to our analysis, more than half of 100 datasets from previous benchmarks come from domains that involve non-I.I.D. setting in real-world deployment. For example, in weather prediction, in applications, usually the task is to predict weather in the future relative to the data available for training. Yet, we do believe that an I.I.D. split of the weather dataset could be useful for comparison of a tabular model’s performance on I.I.D. data. However, we think that the fact that a majority of tabular datasets from the popular benchmarks come from tasks that essentially assume non-I.I.D setting in deployment indicates that researchers are interested in how tabular models work on both I.I.D. data and non-I.I.D. data, however previous benchmarks only focus on one of these fields. In that sense, there is a gap in possibilities of evaluation on current benchmarks, and that is the gap that we meant to address with this paper.

the title and framing of the abstract/introduction could be adjusted such that this work is more about a new, specific benchmark than an alternative or extension of prior benchmarks.

We actively discourage future researchers from treating this benchmark as an alternative to prior benchmarks, describing our work in 4.1 as useful for “further evaluation” of tabular DL methods. However, we believe TabReD could be used as an extension of prior benchmarks. If a new tabular method is developed, it would also be interesting to see how well does it perform on new conditions facilitated by TabReD: time-based split and richer feature sets. If deemed interesting by a researcher, timeshift’s effect could be isolated by also testing a model on random splits of TabReD, which we also provide.

The way in which a model interacts with non-I.I.D. and feature-rich regimes would provide additional information on the model’s limitations. We also do not believe that failure on non-I.I.D. data should negatively influence model’s image in I.I.D. applications, and do not think a negative result on TabReD should prevent a method’s publication, but rather bring more understanding about limits of applicability of the method.

To sum up the whole discussion, our work is not really about time-series and forecasting methods, but about a different application scenario for tabular methods. We believe our benchmark is closer to testing OOD robustness of different methods, like in (Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time, by Huaxiu Yao et al., https://arxiv.org/abs/2211.14238), but we also cover a second uncovered condition: in this benchmark, there are much more features than in previous ones, and we offer random split versions of TabReD to make it possible to isolate this effect from the time-split OOD influence. We think that purely I.I.D. setting is still valid, and do not wish to disparage it. As a result of this discussion, we will add more on the place of I.I.D. vs non-I.I.D. in our paper to the Role and Limitations section. We would also like to draw your attention to the feature-rich character of our data, which marks another shift from previous tabular DL benchmarks (and new results highlighting it in the general response).

Related Benchmarks

We agree that we should have chosen a different wording when describing related benchmarks. We will replace our general terms to refer specifically to tabular deep learning benchmarks.

Finally, the field of time series forecasting offers many more benchmarks for predictive modeling with non-IID data and (slight) temporal shifts.

We would like to differentiate the field of TabReD from the field of time-series. This benchmark explicitly focuses on feature-rich scenarios, in which default tabular data methods perform better than time-series approaches (as we show in Kaggle challenges above).

We sincerely thank you for your review! We think it raised many important points and will help us make our paper better. Below, we address your concerns about our submission.

On I.I.D. vs non-I.I.D. setting:

Thank you for your comments about I.I.D. vs non-I.I.D. settings. We will add more clarifications on the positioning of our work in subsection 4.1.

We agree that benchmarks that cover only I.I.D. setting are valid and useful. We did not intend to give an impression that we treat the choice to limit a benchmark to only I.I.D. setting as a disadvantage.

In non-I.I.D. setting, one could choose to follow one of the two possible approaches: the first is to develop a method that explicitly takes timeshift into account and tries to adjust the predictions with the flow of time, and the second is to just test different approaches that work for I.I.D. data and use them in non-I.I.D. conditions. In the following part of our response, we will demonstrate that on the kind of datasets this paper focuses on, timeshift-agnostic methods are usually used instead of the forecasting methods.

Discussion of timeshift-agnostic vs forecasting methods

We believe there is a separation between forecasting tasks and our non-I.I.D. tabular prediction tasks. For our datasets, we specifically chose datasets with rich features, where the time feature does not overwhelm the importance of other features. In this regard, our task is different from time-series tasks. When looking at the discussion section on Kaggle, we notice that none of the competitions from which we source our data have a competitive solution based on forecasting. Here are the details:

• On Sberbank Housing dataset, the only mention of forecasting is for feature engineering (Forum link). Even in that context, forecasting was not successful and failed to improve performance of the downstream model
• On Homesite Insurance dataset, the only mention of utilizing time-series is again for feature engineering, and again it only gives an insignificant boost to the score of 2e-4 (Forum link). All top decisions mentioned in discussion utilize shift-agnostic models such as XGBoost.
• On the E-Commerse dataset, time and forecasting is not mentioned at all, and the top solution focuses on feature engineering approaches (Forum link).
• On HomeCredit dataset, the mistake in metric choice by hosts resulted in some conditions using unrealistic hacks to win. Nevertheless, in the discussion process, competitors agreed, than when evaluation by pure AUC, feature engineering is more critical (Forum link).

For our newly introduced datasets, no forecasting experiments have proved useful in production, and simple shift-agnostic tabular ML models have proven to be useful in real-world applications scenarios.

Overall, we believe tabular DL/ML and forecasting/time-series are different domains, and we chose to focus on OOD situation in tabular data. In that sense, our paper is closer to Wild-Time (Huaxiu Yao et al.), which also focuses on timeshifting data, albeit in other domains. In that paper, forecasting methods are not used, instead the authors utilize general increased robustness methods, and in this work, we benchmark robustness methods like DFR and CORAL too. Interestingly, in a related field of distributional shift robustness, there are questions about the effectiveness of methods trying to mitigate non-I.I.D. data influence on model’s performance (e.g. In Search of Lost Domain Generalization, Ishaan Gulrajani et al)

Kaggle Data

The paper presents data from Kaggle as the "holy grail" for real-world data and benchmarks.

Specifically, the paragraph about Kaggle in the related work section (Line 113) lacks any critical reflection of data from Kaggle.

We only meant that there are more good datasets on Kaggle than in other sources. We do agree that a general dataset from Kaggle suffers from many issues, and conducted a thorough checking for which datasets we select. We will add a note that we do not refer to all datasets, but only to the selected best ones.

To give some examples, data from competitions on Kaggle can be duplicates, sourced from UCI or OpenML, purely synthetic data, badly preprocessed, or uploaded with a data leak

The discussion section is often useful in identifying these problems. All our selected datasets are free from these issues. The selection criterions are discussed in part 4 and appendix D.

Furthermore, sourcing data from Kaggle competitions and employing the top-performing preprocessing pipelines has two critical problems not discussed in this paper: 1) the pipeline is selected on test performance (leaderboard score) and thus leaking or overfit;

We choose only common-sense approaches to data engineering, avoiding some suspicious choices that may result in overfit (excluding datasets with known leakage, "magic" feature or other hacks). Moreover, no significant meta-overfitting was observed in a large scale study of Kaggle competitons (Rebecca Roelofs et al.) (modulo some outliers, which we do exclude). We also provide our preprocessing code in the repository, and the community is free to experiment with it to find out any suboptimalities in our choices.

2 is specifically bad if, for example, the data only exhibits a crucial distribution shift in the test (or a temporal shift).

Experiments in section 5.4 indicate that temporal shift is present in our provided splits (we've specifically chosen the datasets where the split is possible and the time-frames are realistic in terms of the underlying task)

Questions:

1. In the paper introducing TabZilla (When Do Neural Nets Outperform Boosted Trees on Tabular Data?, by Duncan McElfresh et al., https://arxiv.org/pdf/2305.02997), the authors introduce “TabZilla Benchmark Suit” as a “collection of the 36 ‘hardest’ of the datasets we study”. We chose this version as an official version introduced in the paper.
2. We will add information about the time period from which the dataset comes in the appendix of our paper. All 4 of our newly introduced datasets focus on the regression task. We use RMSE for regression and AUC-ROC for classification. We cannot yet release some information about our datasets due to anonymity policy. However, we will provide the datasheet which we filled-out already.
3. We give links to discussion posts on Kaggle on which we based our data preprocessing in the datasheet. We also provide full preprocessing code for Kaggle datasets.
4. The scores for XGBoost, CatBoost, and LightGBM are in the Table 1 bellow (first three datasets use AUC-ROC, the higher the better, the rest use RMSE, the lower the better). We did not provide ensemble scores originally, because the ensembles of boostings are usually not as effective as ensembles of NNs (Revisiting Deep Learning Models for Tabular Data, by Yury Gorishniy et al., https://arxiv.org/pdf/2106.11959)
5. We will upgrade the appendix to show std’s across seeds. Currently, we would like to draw your attention to subsection C.2, Table 5, which uses multiple comparison statistical test to obtain ranks of different methods.
6. We provide a more succinct version of the figure 2 in the general response.

Table 1. Results for the ensembles of GBDTs:

Thank you for your review! We would like to answer your questions and address the concerns you have about our paper.

No detailed analysis of failure modes for retrieval-based models

We conduct an additional experiment, where we select top-20 important features using XGBoost feature importance and run several methods on them. Here is the table summarizing average improvement over MLP when using all features or 20 features, in percentages:

* NOTE: MLP with all features has an advantage of 5.65% over MLP with 20 features.

With this change retrieval methods represented by TabR start showing performance gains over MLP. Deeper investigation into this could be an interesting avenue for future work.

Given the emphasis on industrial applications, it would be nice to have a sense of how impactful these performance shifts are for the downstream tasks, i.e. can you give us a sense of how much 0.01 RMSE matters for cooking time estimation dataset?

On cooking time dataset, 0.01 RMSE would be a pretty significant improvement, potentially deserving of implementation in production pipeline. It would correspond to about 3% improvement of time estimation in minutes.

it may help strength the argument to a more detailed comparison between the conclusions of your Table 3 to the rankings that would have been generated by running these models on the prior work instead.

We believe that the breadth of comparison in Figure 1 is not a priority. It is addressed by the consistency of our results obtained on the Gorishniy benchmark with the results from the tabular DL literature. For example, retrieval methods (TabR, ModernNCA), embeddings and ensembles show consistent performance across hundreds of datasets in existing benchmarks (the largest one to date is Han-Jia Ye et al., it covers many datasets from Léo Grinsztajn et al. and Tabzilla). Strenghts of the long-training with augmentations is also replicated on a broader set of datasets (Kyungeun Lee et al.).

We would also like to answer your questions.

1. We conducted experiments on different subsamples. Our conclusions hold quite robustly for different subsampling of the full dataset, i.e. average improvement of XGBoost over MLP is 5.21% ± 0.04% on weather dataset (std is taken over subsamples), and 0.34% ± 0.06% on maps-routing dataset.
2. We study the preprocessing pipeline from when the dataset first appeared to its usage in a benchmark, usually a mistake made in this process allows an additional information containing leakage to appear in the data. In some datasets, one of original target variables is added to the features, while some datasets have an answer to one example as a feature in other, and originally used non-random split to avoid leakage, which was changed to random in later literature.
3. These datasets are samples from internal data pipelines from an industrial company. We cannot yet reveal exactly how these features were obtained, partly due to anonymity concerns, however we will provide some additional information upon acceptance.
4. We also focus on richer feature sets, and most datasets where time feature was available have a small number of features.
5. We see two possible use-cases. The first one is to use it to evaluate a method that already shows improved performance on previous benchmarks, to show how well does this method handle a shift to feature-rich and timeshift-aware data. If a model fails to perform on TabReD, one could provide analysis on random splits or 20 feature subsamples we also include in our benchmarks, to better understand the precise reason why a model's performance worsened. The second use-case is for methods that deal specifically with temporal shift or big number of features, since this kind of models can currently be tested only on TabReD.

Thank you for your response and taking the time to answer my questions!

Thank you for the review! We would like to address the weaknesses you pointed out in our paper.

It would be nice to have this information either in Figure 1 or as another column of Table 3 in order to get a better feeling of how the methods perform.

We will add information about the average percentage improvement over MLP in the appendix of the revised paper, shortly.

I am not fully convinced that it is due to 1) temporal data drift or 2) datasets with large number of correlated/uninformative features, as the authors imply.

We provide additional experiments on the effect of large number of features, which helps us demonstrate how both of these points influence a method's performance. We will circle back to this in response to the second point of your second weakness.

I would like to see a table with aggregated results for most methods

To make the presentation of comparison clearer, we plot relative improvements over MLP when using / not using time splits. Results are in the table below:

As expected, performance decreases when using time-based splits, but the impact on the models ranking and relative gap is not clear.

When going from time-split to random splits, the gap between XGBoost and MLP-PLR becomes larger, while the difference between TabR and MLP-PLR becomes smaller.

In addition, most of the methods considered in Table 3 are not included.

For clearer presentation we chose to focus on one method from each of the major categories: retrieval, embeddings, augmentations. Most other methods claim state-of-the-art results on benchmarks on which they were introduced, which often includes one or several of the ones we observe in related work. For this specific in-depth analysis though, due to time and resource constraints, we chose to focus only on method families and not specific methods.

The impact of having datasets with large numbers of correlated/uninformative features is not explored.

Thank you for proposing this useful experiment, which helps us demonstrate the point of our paper! We conduct an additional experiment, where we select top-20 important features using XGBoost feature importance and run several methods on them. Here is the table summarizing average improvement over MLP when using all features or 20 features, in percentages:

* NOTE: MLP with all features has an advantage of 5.65% over MLP with 20 features.

We'll add this result to the PDF shortly. With this change retrieval methods represented by TabR start showing performance gains over MLP. Deeper investigation into this could be an interesting avenue for future work. Furthermore, improved training methodologies for the MLP (aug. rec) are less detrimental in this setup, and they are even helpful for some datasets.

Now, we would like to answer you questions:

1. We somewhat agree, but we want to avoid causing any misunderstanding with oversimplifying in caption, so we explain what we meant in the main text.

2. This statistical test follows "Revisiting Deep Learning Models for Tabular Data" for continuation reasons. For more commonly used statistical significance test you can see a multiple comparison test from appendix C.2, Table 5.