Anytime Neural Architecture Search on Tabular Data

We thank you for your constructive comments and positive feedback. We would like to address your concerns below.

The search space of NAS-Bench-Tabular ... tabular data[1].

Response: Existing studies show that deep neural networks (DNNs) can already achieve state-of-the-art performance on tabular data [4, 5], and the technical challenge is to configure DNNs with the right number of layers and hidden layer sizes for each layer [1]. Therefore, we design a DNN-based search space that comprises up to 160,000 candidate DNNs with an extensive collection of different layer configurations.

As shown in the experiments and visualizations in Section 4.1.1 and Appendix C.3, this search space contains a diverse set of architectures with a wide range of parameter sizes, configurations, and performances, which captures the main properties of DNNs for benchmarking NAS approaches on tabular data.

Most of the candidates... stable the recorded accuracies are.

Response: We have summarized the statistics of our 160,000 trained architectures on the Frappe dataset in the table below.

The very low standard deviation (0.0021) indicates that most architectures closely align with the mean AUC. While the significant gap between the maximum AUC (98.14%) and minimum AUC (85.72%) suggests that the number of hidden neurons in each layer is crucial in determining the architecture performance.

All three datasets are about ... multi-class classification tasks are needed.

Response: As suggested, we evaluated ATLAS on additional datasets: three for multi-class classification and two for regression. The results of multi-class classification are presented in our global response to the Evaluation of ATLAS on more Datasets. The results of the regression are shown in the table below where we have measured the Root Mean Square Error (RMSE).

ATLAS outperforms both XGBoost and CatBoost on Pol datasets with lower RMSE. While for the California Housing dataset, ATLAS achieves comparable results with others.

DNNs are not the dominant methodology ... AutoGluon[3].

Response: As suggested, we have compared ATLAS with XGBoost, CatBoost, AutoSklearn, and AutoGluon on eight datasets. The results are presented in our global response to the Evaluation of ATLAS on more Datasets.

In general, performance proxies ... And what makes ExpressFlow specialize in tabular data?

Response: Existing data type-agnostic TRAILERs such as SynFlow do not perform well on tabular datasets as shown in Table 1 of the original paper. Therefore, We are motivated to propose a new TRAILER specifically designed for tabular data.

To this end, we first characterize the trainability and expressivity of existing training-free metrics initially proposed for image data. Then, we propose a new TRAILER tailored for tabular data based on neuron saliency. This TRAILER characterizes both properties for DNNs on tabular data (see discussion in Section 3.2 and theoretical analysis in Appendix D.3), by considering the activation value of neurons that are basic units to extract features in DNNs and the derivatives of neurons that indicate the importance of features extracted by neurons.

In Equation 2, ... candidate K for each round?

Response: Our ATLAS includes both filtering and refinement phases. For the refinement phase, it uses the Successive Halving Algorithm (SUCCHALF) to identify the best-performing architecture from the $K$ architectures, with each training for at least $U$ epochs.

Indeed, SUCCHALF allocates a different budget of $U * \eta^i$ (with $U$ doubling if $\eta$ = 2) and a number of candidate architectures $\lfloor K/\eta^i \rfloor$ (halving if $\eta$ = 2) for training-based evaluation in the $i$-th round (with i starting from 0). For example, in the first round of evaluation, SUCCHALF trains $K$ architectures, with each training for $U$ epochs. In the second round, the higher-performing $\lfloor K/\eta\rfloor$ architectures are retained, with each training for $U * \eta$ epochs. This process iterates for $\lfloor\log_{\eta}K\rfloor$ training rounds until one single architecture remains.

We hope our responses above have addressed your concerns and can improve your evaluation of our work.

[1] Revisiting Deep Learning Models for Tabular Data [Gorishniy et al., NeurIPS 2021].

[4] Well-tuned simple nets excel on tabular datasets [Kadra et al., NeurIPS 2021].

[5] TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets [Yang et al., NeurIPS 2022].

We appreciate your valuable feedback and constructive comments on helping improve our paper. We would like to address your concerns below.

Benchmark needs to be more diverse ... for an example.

Evaluation of ATLAS on more diverse datasets ... increase my score.

Is ATLAS performant on datasets with less than 1000 data points?

Response: As suggested, we have evaluated our ATLAS on more datasets from the open-source OpenML AutoML Benchmark, and the results are presented in our global response to the Evaluation of ATLAS on more Datasets.

Comparison with conventional tree-based models ... runtime of (2)).

Response: As suggested, we compared ATLAS with XGBoost, assessing their AUC on the Frappe dataset.

Our ATLAS, supporting anytime NAS for tabular data, can operate within a pre-defined time budget. We first set a time budget to run ATLAS to select a well-performing architecture. Then, an additional time budget was allocated to fully train and evaluate this selected architecture. We managed both time budgets to ensure the total time budget was around one hour.

For XGBoost, we also fixed the overall time budget for one hour and tuned the hyperparameter of XGBoost based on training. We employed the same hyperparameter search space as defined in [1] (Table 6).

The results shown in the Table below demonstrate that ATLAS can identify architectures that outperform XGBoost on the Frappe datasets. This confirms the efficacy of ATLAS in finding well-configured DNNs, i.e., determining the appropriate number of hidden neurons for each layer.

Comparison to SOTA for DL for tabular data ... obtain Figure 5 in their paper.

Response: Our ATLAS differs theoretically from TabPFN. TabPFN is a Prior-Data Fitted Network (PFN) capable of approximating probabilistic inference in a single forward pass, it has two limitations: 1) it requires extensive offline training (e.g., 20 hours on 8 GPUs as reported in [3]), and 2) it only scales effectively to small datasets and performs worse when categorical features are presented.

In contrast, our ATLAS employs both training-free and training-based architecture evaluation techniques. When compared to TabPFN, ATLAS demonstrates robust performance across diverse datasets ranging from small to large and consistently performs well when facing either numerical or categorical features, as evidenced in Figure 6 of the original paper.

Table 5 in the TabPFN paper [3] presents the mean AUC of the TabPFN across all 18 datasets from the OpenML-CC18 Benchmark. Due to time constraints, we have evaluated our ATLAS on only eight datasets. For each dataset, we have also measured the total time (Total Cost) it takes to both search for the higher-performing architecture and then train this architecture to evaluate its performance, as shown in our global response to the Evaluation of ATLAS on more Datasets. The results illustrate ATLAS is highly efficient and is competitive with methods detailed in Table 5 of the TabPFN paper.

It would be great if the authors ... and not rename them to TRAILERs.

Response: As suggested, we will refer the TRAILERs to zero-proxies.

In Section 2, third paragraph ... performance evaluation based on the one-shot model weights (e.g. as in DARTS).

Response: As suggested, we will rephrase the sentence to "The performance obtained by the training-based architecture evaluation approaches is typically more accurate".

Most of blackbox algorithms are also anytime algorithms, ... so I would emphasize that instead.

Response: Thank you for your valuable suggestions. We will first rephrase our claims to state 'ATLAS is the first anytime algorithm tailored for tabular data.' Second, we will emphasize that 'ExpressFlow is the first zero-cost proxy designed for tabular data' in our paper.

Why did the authors pick those 3 datasets for their benchmark? Is it because of the large number of training examples?

We explain the reason for choosing the three datasets in our global response to Reasons for Selecting the Three Datasets for NAS-Bench-Tabular.

How many data points were used to compute the rank correlation in Table 1?

Response: In Table 1 of our paper, we present the correlation between each training-free evaluation metric score and the architecture's actual performance (AUC). This analysis uses 160,000 architectures for the Frappe and Diabetes datasets and 10,000 for the Criteo dataset.

We hope our responses above have addressed your concerns and can improve your evaluation of our work.

[1] Well-tuned simple nets excel on tabular datasets [Kadra et al., NeurIPS 2021].

[2] Revisiting Deep Learning Models for Tabular Data [Gorishniy et al., NeurIPS 2021].

[3] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second [Noah et al., ICLR 2023].

We are very glad to know that our response has addressed most of your concerns with clarity. We would also like to thank you for raising your score for our paper. As per your suggestion, we will update the main paper with the results from our global response, along with additional evaluations on the OpenML-CC18 benchmarks. Additionally, to avoid confusion and for clearer representation, we will rename "NAS-Bench-Tabular" to "NAS-Bench-TD". Thank you again for your constructive comments.

We highly appreciate your feedback. However, we would like to point out some misunderstandings about our work.

The development of ATLAS ... of NAS in real-world scenarios where time and resources are often limited.

Response: Thank you for appreciating the contribution of our NAS tabular data benchmark (NAS-Bench-Tabular). We will make it publicly available to facilitate further research and development in the area of NAS for tabular data.

A limitation of the paper ... and more complex tabular datasets that are commonly found in practice.

Response: The datasets utilized in our work are real-world tabular data. Specifically, we select three datasets from different application domains: app recommendation, healthcare analytics, and CTR (e-commerce) prediction. The statistics of the datasets are summarized in the Table below:

Notably, for the three datasets, there are 10, 43, and 39 columns, corresponding to 5,382, 369, and 2,086,936 unique numerical and categorical features respectively, which are high-dimensional.

To further validate the generalizability and effectiveness of our proposed ATLAS, we have conducted experiments on an additional eight datasets from the open-source OpenML AutoML Benchmark.

The evaluation results, as presented in our global response to the Evaluation of ATLAS on More Datasets, confirm the effectiveness and generalizability of our proposed ATLAS.

The missing significant baselines ... NAS methods may provide in tabular data applications.

Response: We would like to clarify that the scope of this paper is to propose a novel anytime NAS approach tailored for tabular data, rather than to design a new architecture that outperforms established architectures such as TabTransfer or FT-Transfermor. As a result, we mainly pick baselines from the state-of-the-art NAS approaches designed for tabular data. Specifically, we utilize TabNAS [1] and training-based NAS approaches, i.e., RE-NAS, for comparison as shown in Table 6 and discussed in Section 4.3 of the original paper.

To further demonstrate the practical advantages of our ATLAS, we have measured its balanced accuracy on two datasets from the OpenML AutoML Benchmark and compared ATLAS against XGBoost, TabTransformer, and FTTransformer. The results shown in the table below indicate that the selected architecture of ATLAS is comparable to both transformer-based architectures and XGBoost.

Both TabTransformer and FTTransformer are designed to capture complex relationships in tabular data through self-attention mechanisms, and thus they perform consistently well on two datasets. Notably, our ATLAS effectively determines the number of hidden neurons for each layer of an MLP, thereby making it competitive with other methods. The experimental results align with conclusions drawn from FTTransformer [2]: specific tuning can make simple models like MLPs competitive.

We hope our responses above have addressed your concerns and can improve your evaluation of our work.

[1] TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets [Yang et al., NeurIPS 2022].

[2] Revisiting Deep Learning Models for Tabular Data [Gorishniy et al., NeurIPS 2021].

We hope our clarifications and new experiments have addressed your concerns. We would highly appreciate your consideration for re-evaluating your initial rating. We look forward to any further feedback you may have.

We thank you for your constructive comments and positive feedback. We would like to address your concerns below.

The NAS tabular data benchmark ... It would be great if the authors released the dataset publicly.

Response: Thank you for appreciating the contribution of our NAS tabular data benchmark (NAS-Bench-Tabular). We will make it publicly available to facilitate further research and development in the area of NAS for tabular data.

The performance evaluation is conducted using the limited search space ... and other components of MLP, is unclear.

Response: Existing studies show that deep neural networks (DNNs) can already achieve state-of-the-art performance on tabular data [1, 2], and the technical challenge is to configure DNNs with the right number of layers and hidden layer sizes for each layer [3]. Therefore, we design a DNN-based search space that comprises up to 160,000 candidate DNNs with an extensive collection of different layer configurations.

As shown in the experiments and visualizations in Section 4.1.1 and Appendix C.3 of the original paper, this search space contains a diverse set of architectures with a wide range of parameter sizes, configurations, and performances, which captures the main properties of DNNs for benchmarking NAS approaches on tabular data.

Could you comment on the applicability ... search space used in this paper?

Response: Our proposed method, ExpressFlow, is transferable and can be applied to other search spaces, such as NAS-BENCH-101 and NAS-BENCH-201 [4], which are designed for vision tasks.

However, ExpressFlow is specifically tailored for tabular data, as it is based on neuron saliency that captures the complex and non-intuitive relationships among input features in this data type.

Consequently, while demonstrating higher correlation values on tabular datasets (as shown in Table 1 of the original paper), ExpressFlow achieves only the second-highest average rank in terms of correlation value for image data, as illustrated in Table 15 of Appendix G.1.

Is it possible to apply the proposed method ... additional budget will be available after the algorithm starts?

Response: Thank you for the insightful comments and suggestions. We agree that this is a very practical situation.

Our NAS approach is anytime and thus can readily adapt to this situation by continuing to search for better-performing architectures with the current available time. Specifically, our approach includes two distinct phases, namely, the filtering phase and the refinement phase, along with a coordinator. If the additional budget is available during the filtering phase, the coordinator can dynamically allocate more time to both phases, which allows for exploring more architectures in the filtering phase and exploiting more architectures in the refinement phase. Conversely, if the additional budget is available during the refinement phase, all budgets are allocated to this phase, and thus each architecture will be trained over more epochs during the successive halving process.

We hope our responses above have addressed your concerns and can improve your evaluation of our work.

[1] Well-tuned simple nets excel on tabular datasets [Kadra et al., NeurIPS 2021].

[2] TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets [Yang et al., NeurIPS 2022].

[3] Revisiting Deep Learning Models for Tabular Data [Gorishniy et al., NeurIPS 2021].

[4] Nas-bench-suite-zero: Accelerating research on zero cost proxies [Krishnakumar et al., NeurIPS 2022].