Dear Reviewer B7Cf:

Thank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.

Q1: Shifted features vary across different examples.

We appreciate your feedback on our experimental design. In our random shift experiments, to evaluate model performance under n/N% feature shift (where n is the number of shifted features and N is the total feature count), we follow this protocol:

1. Randomly sample min(10,000, C(N,n)) distinct feature combinations for each shift magnitude.
2. Remove each selected combination during testing to assess performance.
3. The performance for a given n/N% shift is the mean across all combinations.

Q2: Considering cases with different types of feature shifts

Regarding this question, we would like to clarify two potential interpretations:

• Different types of (feature shifts): Please refer to our response to Reviewer Q7p8 Q1. The first leaderboard provides a comprehensive evaluation of model robustness across various types of feature shifts.
• Different types of (feature) shifts: We conducted evaluations of CatBoost performance on a single shift, categorized by: Categorical, Numerical and Boolean features. Results demonstrate that model performance exhibits the highest sensitivity to Categorical features, followed by Boolean features while showing the lowest susceptibility to Numerical features. The complete experimental results are available in the link. |Metric|Raw|Categorical|Numerical|Boolean| |-|-|-|-|-| |Acc|0.792|0.769|0.787|0.778|

Q3: Feature-increment scenario.

Please refer to our response to Reviewer Q7p8 Q2.

Q4: More datasets.

Due to time and computational constraints in the rebuttal phase, we couldn't complete evaluations on larger benchmarks. However, following OcTree, we conducted random shift experiments on two high-dimensional datasets (madelon and nomao) and present model performance under 10%, 20%, ..., and 100% feature shift degrees, limited by space. This link has detailed results.

The table demonstrates that as the degree of feature shift increases, the model performance decreases significantly. Please refer to our response to Reviewer Q7p8 Q1 for further explanation.

Q5: Imputation methods.

We appreciate your suggestions. As Reviewer Q7p8 and WumN noted, our paper is a benchmark study and needn't propose new methods. Thus, we did not explore AutoFE in feature shifts. Additionally, our focus on evaluating model robustness meant we did not extensively cover advanced imputation methods, as explained in Section 3.3.

In light of your proposal, we posit that AutoFE can address feature shift scenarios through two approaches. Detailed results can be found in the link.

• Imputation: CAAFE and OcTree generate rules for imputing specific features, while OpenFE's generated features do not match the originals. We tested single shifts on the heart dataset using CAAFE and OcTree for imputation. Results show that LLM-based AutoFE can effectively generate matching features, enhancing model robustness compared to mean imputation. |Model|Raw|CAAFE|OcTree| |-|-|-|-| |CatB|0.846|0.852|0.850| |PFN|0.859|0.869|0.870|

• Generation: We used AutoFE to generate new features to offset the impact of missing original features on model performance. Tests on the heart dataset with random shifts via OpenFE, CAAFE, and OcTree (with the number of generated features matching the original dataset) showed that LLM-based AutoFE has significant potential in feature shift scenarios. |Model|Raw|OpenFE|CAAFE|OcTree| |-|-|-|-|-| |CatB|0.710|0.849|0.854|0.849| |PFN|0.763|0.859|0.867|0.865|

As the importance of features increases, both model performance declines, further corroborating Observation 2 of TabFSBench. Please refer to our response to reviewer wA1o Q7 on other imputation methods.

Q6: Typos.

We will make revisions in the final version.

Q7: Questions For Authors.

Feature importance, calculated via PCC, correlates strongly with the impact of feature absence on performance. Higher importance results in greater performance decline, indicating that model dependency on features is consistent across closed environments and feature shift scenarios. This stability is due to the models' robust feature selection and weight allocation mechanisms. For more details, refer to our response to reviewer wA1o Q6 and the link.

Dear Reviewer wA1o:

Thank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.

Q1: Weakness&Suggestion 1

We adopt notations used in our paper and will provide a more detailed explanation in the final version.

• Distribution Shift: During the training phase and the testing phase, the feature sets remain unchanged, i.e., $C_{train} = C_{test}$. However, there are three types of distribution shifts of the samples themselves: $p(x)$, $p(y|x)$, and $p(x|y)$, where $x \in \mathcal{X}$ and $y \in \mathcal{Y}$.

• Feature Shift: During the training phase and the testing phase, the feature sets change while the data distribution of samples has no shift.

  • Increment: $C_{train} \subseteq C_{test}$.
  • Decrement: $C_{test} \subseteq C_{train}$.

Q2: W&S 2.

Please refer to our response to Reviewer Q7p8 Q1.

Q3: W&S 3.

We implement adaptive hyperparameter optimization based on the Optuna framework and following previous studies[1], fixing the batch size at 1024 and conducting 100 independent trials through train-validation splits to prevent test set leakage, with the best-performed hyperparameters fixed during the final 15 seeds.

Q4: W&S 4

Please refer to our response to Reviewer B7Cf Q4.

Q5: W 5

Since our paper focuses on evaluating the robustness of tabular machine learning models in feature-shift scenarios, as Reviewer Q7p8 and WumN noted, our paper is a benchmark and doesn't propose new methods. Therefore, we did not propose a new method. However, through our response to reviewer B7Cf Q5, we explored the potential of AutoFE in addressing the challenges of feature shifts.

Q6: The justification for the Pearson correlation is weak.

We calculated Kendall's $\tau$ coefficient among Pearson, Spearman, SHAP, and mutual information to assess the consistency of feature importance rankings. The table showed a high degree of consistency across these metrics:

Despite Spearman's high consistency ($\tau$ = 0.61), We chose Pearson for its widespread use and interpretability. The minor difference in consistency between Pearson and Spearman does not affect the conclusions of the analysis. Detailed data are provided in the link. We will further refine related experiments in the final version.

Q7: The claim that LLMs show potential in feature shifts is not rigorously validated.

We evaluated LLM performance in scenarios without missing value imputation. Experimental results demonstrate that LLM can not only handle feature shifts but is also more robust than tree-based models. Please refer to our response to reviewer 8NVr Q5. Additionally, we tested the performance of LLM-based AutoFE in handling feature shifts. Please refer to our response to reviewer B7Cf Q5.

Q8: Observation 2 lacks theoretical justification.

This observation is drawn from experimental results, thereby limiting the scope of the analytical section. To further investigate this observation, we conduct a theoretical justification.

A high PCC signifies a strong feature-target relation, enabling the feature to provide substantial information increment and enhance model performance. Thus, removing important features leads to a significantly greater performance degradation than removing unimportant ones. Additionally, the model depends strongly on highly correlated features during training, and their absence markedly deteriorates performance (see single-shift experiment results).

We have already pointed out the limitations regarding the impact of interactions among input features on model performance in lines 433-435 in the Conclusion of our paper. Subsequently, we will conduct an in-depth exploration of this. For additional explanations regarding this observation, please refer to our response to Reviewer 8NVr Q2.

Q9: Not cite all relevant work.

Thank you for your suggestion. Although existing Heterogeneous Domain Adaptation (HeDA) methods have achieved significant progress on feature shift of images, tabular data presents a fundamentally different pattern. The inherent structure of it presents challenges when attempting to directly implement HeDA on such datasets. Moreover, our review of the literature reveals that what is often referred to as "feature shift" in many papers is essentially a form of distribution shift. For example, [2] regards covariate shift as feature shift. Other related works on feature shift discussed in our paper, please refer to our response to Reviewer 8NVr Q10.

[1] Liu, Si-Yang, et al. TALENT: A Tabular Analytics and Learning Toolbox. arXiv preprint arXiv:2407.04057, 2024.

[2] He, Huan, et al. Domain adaptation for time series under feature and label shifts. International Conference on Machine Learning, 2023.

Dear Reviewer 8NVr:

Thank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.

Q1: It is difficult to rely on noisy results without clear trends.

We would like to clarify that |$\rho$| greater than 0.7 can be considered as a moderate linear correlation[1]. Tableshift[2] figure 5 also draws a conclusion of linear correlation based on $\rho$ value of 0.7. In our paper, $\rho$ value in Figure 3 is 0.7405, and $\rho$ value in Figure 4 is 0.6. Therefore, despite the presence of noisy data, relevant conclusions can still be drawn based on $\rho$ values. We will add the evaluation results of more datasets to mitigate the noise.

Q2: why are there multiple trajectories in Figure 3?

The reason for multiple trajectories in Figure 3 is that each trajectory consists of points derived from the same dataset. During the fitting process, the results of all datasets were combined, resulting in each dataset forming a distinct trajectory. It is worth noting that these trajectories reflect the relationship between feature importance and model performance across different datasets. Specifically, each dataset's trajectory demonstrates how its specific feature importance influences the model's performance.

To better support our conclusions, this link provides correlation and accuracy plots for 12 different datasets. Below are $\rho$ values for the relationship between correlation and accuracy for each of the 12 datasets.

It indicates that although levels of feature importance vary across different datasets, they all support our research conclusion, namely, that there is a significant linear correlation between feature importance and model performance. These trajectories further confirm the generality and reliability of our conclusion.

Q3: Incorporate more high-quality tables.

Please refer to our response to Reviewer B7Cf Q4.

Q4: API design

We have already provided the --export_dataset option in our paper. Users can set this option to True to export specific versions of the datasets they wish to evaluate (e.g., single-column missing, missing to a certain extent, and all possible missing scenarios).

The README.md in Supplementary Material already explains how users can add new datasets and new models. We will further elaborate on the code functions in the final version.

Q5: Major comment 3&4.

1. Reviewer B7Cf has mentioned that "The mean-value imputation strategy used is a common approach and therefore is a valid choice." We also found that benchmark datasets such as Talent[3] also use mean imputation as a method for handling missing values.

2. We compared the performance of various models using their own imputation methods, random imputation, and mean imputation. We present the average performance of several representative models under different imputation methods. |CatBoost|NAN|Mean|Random| |-|-|-|-| |0%|0.845|0.851|0.818| |9%|0.820|0.826|0.807| |18%|0.792|0.801|0.767| |27%|0.763|0.775|0.739| |36%|0.733|0.747|0.702| |45%|0.702|0.716|0.665| |54%|0.670|0.682|0.630| |63%|0.637|0.643|0.608| |72%|0.600|0.596|0.574| |81%|0.538|0.541|0.502|

Our experiments show that the model performs best under mean imputation, and its performance declines under all three types of imputation methods, indicating that the model still faces challenges from feature shift. Detailed data are presented in this link and we strongly recommend that you view this link.

Q6: Major comment 5.

Please refer to our response to Reviewer WumN Q2.

Q7: Minor comment 1&2 & typos ec.

We will make revisions in the final version.

Q8: Minor comment 3.

The purpose of the single-shift experiment is to test the impact of the absence of features with different importance levels on model performance. "Removing one feature randomly" is the experimental setting for a feature shift degree of $\frac{1}{n}$ (n = number of columns) in our random shift experiment.

Q9: Minor comment 4.

Please refer to our response to Reviewer wA1o Q3.

Q10: Minor comment 5.

We have already cited the paper you mentioned on lines 683-684 of Section A.3 on page 13 of our submitted paper.

[1] Iversen, Gudmund R, et al. Statistics: The conceptual approach. Springer Science & Business Media, 2012.

[2] Gardner, Josh, et al. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 2023: 53385-53432.

[3] Liu, Si-Yang, et al. TALENT: A Tabular Analytics and Learning Toolbox. arXiv preprint arXiv:2407.04057, 2024.

Dear Reviewer WumN:

Thank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.

Q1: The authors should provide datasets that naturally contain feature shift issues rather than simply analyzing performance on synthetic datasets.

Regrettably, there currently exists no dataset specifically designed for feature shift, unlike Tableshift [1] which was developed for distribution shifts. However, we have preliminarily constructed a feature-shifted dataset based on the Heart dataset. Given that different features in the original dataset require distinct measurement instruments, we categorized the features into three groups: basic features, electrocardiogram (ECG) features, and exercise stress test features.

In the constructed feature-shifted Heart dataset, both the training set and the test set step 0 contain all features. However, patients in the test set step 1 lack ECG measurements, resulting in the absence of RestingECG and ST_Slope features. Similarly, patients in the test set step 2 did not undergo an exercise stress test, leading to the absence of ExerciseAngina and Oldpeak. A subset of examples is provided in this link.

Additionally, we evaluated CatBoost on this dataset:

Note that for meaningful evaluation of feature-shifted datasets, models must be assessed under specific partitioning schemes. Applying the four experimental settings proposed in our paper would undermine the unique characteristics and practical relevance of such datasets. In the final version, we will include a comprehensive evaluation of the feature-shifted Heart dataset and will open-source the dataset for broader research use.

Q2：What is the performance upper bound for each dataset when the feature shift issue occurs?

We evaluated the performance upper bound of the models by training directly on the shifted data and assessing their performance on a held-out test set. Due to time constraints, we did not test the performance upper bounds of LLMs and tabular LLMs. The detailed performance for all evaluated models on 9 datasets is provided in this link, with the average performance upper bound and feature-shift performance on the Heart dataset presented here.

The results demonstrate that, under the same degree of feature shift, models trained on the original dataset exhibit inferior performance compared to those trained directly on the shifted dataset. This indicates that although training on the original dataset provides more comprehensive information, feature shift significantly compromises model robustness, resulting in performance degradation relative to models trained on the shifted data. Furthermore, as the degree of feature shift continues to increase, the performance gap between original and shifted datasets also becomes progressively larger, indicating a progressively diminishing robustness of the model.

Q3：Could the authors provide some examples of feature shift issues occurring in the real world?

Section 2.2 of our paper employs forest disease monitoring as a case study to demonstrate how sensor degradation leads to a reduction in available features. As further evidenced by the designed heart dataset in Q1, incomplete medical examinations may result in missing diagnostic indicators (features) due to the absence of specific equipment.

The feature shift phenomenon also manifests prominently in financial and transportation domains:

• Finance: Stock prediction models trained on comprehensive features (e.g., financial ratios, macroeconomic indicators) may encounter missing features (e.g., market sentiment indices) during real-world deployment due to unforeseen events.

• Transportation: Accident prediction models relying on features like road conditions and weather data may experience partial feature absence caused by sensor failures or insufficient data collection.

[1] Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 2023: 53385-53432.

Dear Reviewer Q7p8:

Thank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.

Q1: I hope the authors can release a leaderboard for the tabular data learning methods on this benchmark.

We have currently implemented a leaderboard for TabFSBench and incorporated additional evaluation results for TabPFN v2. This link provides access to the TabFSBench homepage we designed. Upon completion of the paper review process, public access will be granted. Furthermore, we will continuously update the benchmark with the evaluation results of new models. Researchers are also cordially invited to contribute their own evaluation results on additional datasets or models. Additionally, we'll be expanding TabFSBench in the future from below two parts:

1. For the dataset selected in TabFSbBench.

We selected 12 datasets from Grinsztajn[1] and TabZilla[2]. They exhibit substantial heterogeneity in scale, domain, and task characteristics, deliberately encompassing diverse potential scenarios of feature shift. To comprehensively evaluate the challenges posed by feature shifts, we have implemented four distinct experimental configurations that collectively enhance both the breadth and depth of our benchmark assessment. Multiple experimental repetitions were conducted to mitigate stochastic variability.

Due to rebuttal time constraints, we were unable to review the large table benchmark in a short period of time. We will incorporate evaluation results based on the Grinsztajn benchmark into the final version.

2. For the future work for TabFSBench.

• We have established and maintained a project homepage and a comprehensive rank leaderboard. We continuously update the performance evaluation results of various newly released models or datasets (for example, the benchmark test data of the recently included TabPFN v2 model have been incorporated), ensuring that the research community can access the latest evaluation information in a timely manner. This homepage will be made open source in the subsequent phase.

• To enhance the scalability of cross-model comparison research, TabFSBench has been specifically designed with a feature-shift dataset export module. Through command-line parameter settings, this module can export datasets under various feature-shift scenarios, thereby effectively supporting researchers in conducting evaluations on their own models (for specific implementation details, please refer to our response to Reviewer 8NVr Q4).

• In the anonymized code repository provided in Supplementary Materials, TabFSBench has implemented a plug-and-play interface for importing new datasets and new models. This enables researchers to conveniently test their custom datasets or models within the TabFSBench framework (for specific implementation details, please refer to our response to Reviewer 8NVr Q4).

• We encourage and actively accept researchers to provide us with their own evaluation results. Subsequently, we will also be committed to constructing feature-shift datasets with practical significance and opening up both public and private evaluation rank leaderboards. This will further promote the research and evaluation of the feature-shift scenario within the community (for other relevant content, please refer to our response to Reviewer WumN Q1/3).

Q2: In the paper, the benchmark doesn’t consider the feature increment.

Section 2.2 of the paper indicates that this study primarily focuses on the impact of feature shifts on model performance. However, tabular machine learning models evaluated in this work are inherently incapable of handling newly added features in feature-increment scenarios, as they require consistent input and output dimensions. Consequently, in such scenarios, these models automatically disregard the new features while maintaining their original performance.

In the future work, we plan to extend the evaluation scope of TabFSBench by incorporating additional datasets and model evaluations. Specifically, we will include specialized models designed for feature increment scenarios (e.g., [3][4]) to assess their performance improvements under such conditions.

[1] Léo, Grinsztajn, et al. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 2022: 507-520.

[2] McElfresh, Duncan, et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems, 2023: 76336-76369.

[3]Zhang, Zhen-Yu, et al. Learning with feature and distribution evolvable streams. International Conference on Machine Learning, 2020.

[4]You, Dianlong, et al. Online learning for data streams with incomplete features and labels. IEEE Transactions on Knowledge and Data Engineering, 2024.