Interpretable Mesomorphic Networks for Tabular Data

We thank the reviewer for providing valuable feedback. Below we address the concerns raised by the reviewer:

• Regarding: “Weakness 1

  Supposing a single input example $x$ with $d$ features, one would need to project each of the $d$ features to a fixed dimensionality to represent tokens, in this scenario the context length would be the number of features $d$. Then, most commonly, related work in the domain appends a classification token [1]. Strictly speaking, a transformer consists of projection weight matrices, like $w_q$, $w_k$, and $w_v$. The attention matrix of the transformer gives a similarity score to the different $w_q$ and $w_k$ representations, and not to the original features.

  In the transformer layer, one would use the value of the classification token which would be a combination of the different $w_v$ of the other feature representations based on the similarity scores. This would be a combination of the other feature representations which after the transformer layer would then pass through an output layer to generate the class predictions as in [1]. On the contrary, our work does not use a combination of the different feature representations to generate the output. One could consider the output weights to the linear model generated from our hypernetwork to be an analogous operation.

  In the case of attention over instances, it would be different from our method since our method takes as an input only one example at a time.

  TabNet has a different architecture that employs sequential attention, it features a series of steps where it learns sparse matrices $M$ of the different original features for every decision step. Based on the mask of a step, the decision step will have an impact on the final output. The masks are then combined based on the decision step importance $\eta$ to the final output. On the contrary, our work does not feature multiple decision steps, and after generating the weights, the output is linearly dependent on them.

• Regarding: “Weakness 2”

  The aforementioned scenario by the reviewer would not impact our method, as the target linear model is generated per example. The hypernetwork will learn to fit every point and it will retain a global accuracy as shown in Section 2.4. The interpretability experiments in Section 5, Hypothesis 3 include piece-wise functions, and based on the results we argue that our method has competitive performance with the other interpretability baselines.

  The hyperplane based on a given example generates the weights which combined with the example features provide the impact of the individual features. Zeroing out an input feature to the hyperplane would generate predictions for a different example. The regularizer in the loss helps to induce sparsity in the generated weights, however, it only marginally improves the prediction quality of the interpretability. We have experimented with removing the regularizer in the loss and we observed that the predictions from our method were robust.

• Regarding: “Question 1”

  The reviewer is correct in his\her understanding. We would like to kindly point the reviewer to Line 342 in the main manuscript, where we link to Appendix A, where we have provided a proof-of-concept example on a vision task.

• Regarding: “Question 2”

  The coefficients of the hyperplane from the Taylor expansion of the first order are given by the derivative of the output with regard to the example features. As for IMN, we perform the multiplication of the derivative with the corresponding feature and we compare both methods over the interpretable benchmark. We provide the results below:

  Dataset	Metric	IMN	Taylor
  	Faithfulness	0.987	0.989
  	ROAR Faithfulness	0.639	0.558
  Gaussian Linear	Infidelity (Lower better)	0.007	0.009
  	ROAR Monotonicity	0.785	0.757
  	Shapley Correlation	0.999	0.999
  	Faithfulness	0.621	0.427
  	ROAR Faithfulness	0.027	0.050
  Gaussian NonLinearAdditive	Infidelity (Lower better)	0.018	0.030
  	ROAR Monotonicity	0.637	0.625
  	Shapley Correlation	0.741	0.618
  	Faithfulness	0.841	0.823
  	ROAR Faithfulness	0.404	0.458
  Gaussian Piecewise	Infidelity (Lower better)	0.008	0.082
  	ROAR Monotonicity	0.682	0.605
  	Shapley Correlation	0.875	0.560
  Wins		11	3

  As observed, IMN outperforms the Taylor approximation and wins 11-3 in the provided benchmark.

We believe to have correctly addressed all the questions raised by the reviewer. If the reviewer has additional questions, we are more than happy to answer them.

[1] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, 18932-18943.

We would like to thank the reviewer for the valuable feedback. Below we will address the concerns raised by the reviewer:

• Regarding: W1

  Implementation:

  We would like to point out to the reviewer that we reuse simple feed-forward backbones from previous work [1]. The only extra operation is implemented by a single line (Line 58 in module hypernetwork.py at the provided code).

  Training:

  Our code offers a fit/predict interface similar to the well-known and widely used scikit-learn library [2] which should facilitate usage for the practitioners. Additionally our method uses the stochastic gradient descent training procedure which is the standard. We would like to remind the reviewer that our approach is end-to-end, in this regard there is only one model class and not a series of components that have to be fit individually.

  Lastly, we would like to point the reviewer to Table 2 in the manuscript, where as observed, our method has a faster median training time compared to the tabular ResNet. (As a result of the epoch being a hyperparameter, which translates to IMN having a faster convergence)

  Deployment:

  After training the model, a practitioner would only need to instantiate the class and load the weights. We kindly point the reviewer to Table 2 in the main manuscript, where as observed, IMN has a comparable inference time compared to the tabular ResNet. IMN can be run on the CPU and for inference/deployment the model can additionally be quantized.

  Based on the above information, we believe the aforementioned aspects are trivial for our proposed method.

• Regarding: W2

  When HPO is used:

  To verify that IMN does not need extensive hyperparameter tuning and computational resources when HPO is applied, we compare the time it takes IMN to find the incumbent configuration versus the time it takes the plain counterpart, the tabular ResNet. We provide a comparison of the incumbent performance of both methods versus the number of trials. As in the main manuscript (Table 4), we select 3 datasets with distinctive characteristics (number of instances/ nr features): Credit-g (1000/21), Adult (48842/15), and Christine (5418/1637). We present the results in Figure 2 at the attached document to the Global Response, where as observed, it takes only a few trials to find a well-performing hyperparameter configuration for both IMN and the tabular Resnet. Additionally, it takes IMN the same number of trials as the tabular ResNet to find the optimal hyperparameter configuration.

  When HPO is not used:

  We would kindly point the reviewer to Line 305, “Hypothesis 1 and 2 are valid even when default hyperparameters are used, for more details we kindly refer the reader to Appendix B”.

• Regarding: W3

  We agree with the reviewer that other data modalities are important, however, we believe tabular data are also very important and ubiquitous [3], due to the numerous application domains involving tabular data. In that context, our focus is on tabular data which in contrast to the other mentioned modalities is structured. We believe investigating other modalities merits a separate investigation of its own and falls outside the scope of our work.

  Despite the above, we would like to point the reviewer to Line 342 in our main manuscript: “Although our work focuses on tabular data, in Appendix A we present an application of IMN in the vision domain.”

• Regarding: Q1

  We would like to point out to the reviewer that we run our method on the AutoML benchmark [4], a benchmark widely used by the community that features a diverse set of real problems. This includes large-scale tabular datasets from real-world applications.

  Moreover, Table 2 provides time information regarding the time complexity of our method and the baselines over all datasets. Additionally, in Table 4, we provide a more specific analysis for hand-picked datasets that have different characteristics showcasing the scalability of IMN regarding inference time.

  Lastly, as an example, on the largest dataset (airlines, circa 583k training instances) IMN takes only 6.9 hours for training and has an inference time of 0.195 seconds.

• Regarding: Q2

  We investigate the sensitivity of IMN and CatBoost with regard to the hyperparameter configuration used. For every task included in our experiment, we generate a distribution of the validation AUROC performances for all the explored hyperparameter configurations per method.

  Figure 3 in the attached document to the Global Response presents the results, where as observed, IMN has a comparable sensitivity to CatBoost with regard to the controlling hyperparameter configuration. Moreover, in the majority of cases, the IMN performance does not vary significantly.

• Regarding: Q3

  We would like to kindly point the reviewer to Hypothesis 2, 3, and 4, where we compare IMN with TabNet, an interpretable method that employs attention. IMN manages to outperform TabNet: in terms of performance with a statistically significant difference (Hypothesis 2, Figure 4), in terms of training/inference time (Hypothesis 2, 3 and Table 2, 4), in terms of local interpretability (Hypothesis 3) and in terms of global interpretability (Hypothesis 4, Figure 6).

  Our results are consistent with the results of previous work [5].

We believe we have clarified all the concerns raised by the reviewer. We would kindly ask the reviewer to increase the score and recommend acceptance based on the provided clarifications.

References:

[1] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

[2] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830.

[3] Van Breugel, B. & Van Der Schaar, M.. (2024). Position: Why Tabular Foundation Models Should Be a Research Priority. Proceedings of the 41st International Conference on Machine Learning in Proceedings of Machine Learning Research.

[4] Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., & Vanschoren, J. (2019). An open source AutoML benchmark.

[5] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.

I appreciate your thorough response to my comments. While your clarifications address some of the concerns, I still have reservations about the interpretability study and the overall experimental design.

Interpretability study limitations Your interpretability study (hypothesis 4) is based on a relatively small dataset (Census with only 10+ features) without noise data. The additional study on mushroom edibility prediction also uses a well-studied dataset where high performance (F1 score of 0.99) is easily achievable. These choices in experimental datasets limit the generalizability of your interpretability claims.

Real-world application cases The paper would be significantly strengthened by including detailed case studies from real-world applications (not just well-studied datasets). These could demonstrate IMN's effectiveness and interpretability in practical scenarios such as financial risk assessment, healthcare predictions, marketing analytics, or HR decision support.

Regarding your dataset selection To increase confidence in IMN's capabilities, it would be valuable to see results on a wider range of datasets, especially those with more complex feature interactions, higher dimensionality, and the presence of noise.

While I acknowledge the positive scores from other reviewers, these remaining concerns prevent me from increasing my score.

We thank the reviewer for the valuable feedback. Below we will address the concerns raised by the reviewer:

• Regarding: “The authors restrict their baseline to interpretable classifiers. There are a lot of recent studies on tabular data that authors should compare with.”

  The main focus of our work lies in interpretability. We compare with state-of-the-art tabular models (Hypothesis 2), to verify that our method is comparable in accuracy and it does not suffer a significant degradation in accuracy. Based on recent work [3] which extensively compares tabular models on a wide range of datasets (circa 176 datasets), we selected the top-performing methods for different model types (Neural Networks and Gradient Boosted Decision Trees).

  We believe the existing set of included baselines is representative of the most frequent models used in the tabular domain and which achieve competitive performance.

  Moreover, we add two additional baselines to our experimental protocol, DANet [1] and HyperTab [2] as the reviewer suggested. The results are presented in Figure 1 at the Global Response. We describe the results in more detail in our next response to the reviewer.

• Regarding: “This is not the first work on uisng hypernetworks on tabular data. The authors of [1] use hypernetworks to build an ensemble of neural networks. The auhtors should at least refer to this work in their paper”.

  We thank the reviewer for providing a valuable reference. Following the reviewer’s suggestion, we additionally add HyperTab to our experiments. We provide the results on our benchmark for all methods in Figure 1, where as observed, HyperTab achieves a similar performance to IMN, with a slightly lower rank.

  To validate the HyperTab results, we additionally compare all methods for datasets that have circa 1k instances and less (5 datasets), the results are presented in the table below:

  Method	Average Rank
  TabNet	7.0
  DANet	5.4
  IMN	4.0
  Random Forest	3.2
  TabResNet	3.0
  CatBoost	2.8
  HyperTab	2.6

  The results are consistent with the results from the original authors [2], where the authors advocate that “HyperTab performs strongly for small datasets however, it is comparable with other methods for medium and large-scale datasets”. This in turn validates our results.

  We will make sure to include the provided reference in the related work and describe the differences between HyperTab and ours. For e.g.:

  • Our work takes as input the full view of original features, while HyperTab takes as input a binary mask depicting the subset of selected original features.
  • Our work makes a single forward pass through the hypernetwork, while HyperTab does multiple forward passes through the hypernetwork to build an ensemble of target networks.
  • The target network of IMN is a single output layer of 1 unit in the case of a binary classification task and $k$ units in the case of multi-class classification where $k$ represents the number of classes. While HyperTab has a target network of a hidden layer of multiple units and an output layer (2 layers in total).
  • The referenced work focuses on accuracy while ours focuses on interpretability.

• Regarding: “The global classification rule is given by the hypernetwork. How the classification changes for nearby points. The authors could illustrate it with an example.”

  We would like to kindly point the reviewer to Section 2.4, Figure 2 left, where we have provided an example of how the global classification boundary looks based on the hypernetwork prediction as the reviewer requests. The section additionally provides information on the classification accuracy of the locally generated hypernetworks regarding neighboring points.

We believe to have addressed the concerns raised by the reviewer and we welcome any other questions/uncertainties that the reviewer might have.

[1] Chen, J., Liao, K., Wan, Y., Chen, D. Z., & Wu, J. (2022, June). Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 4, pp. 3930-3938).

[2] Wydmański, Witold, Oleksii Bulenok, and Marek Śmieja. "Hypertab: Hypernetwork approach for deep learning on small tabular datasets." 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 2023.

[3] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.

We would like to thank the reviewer for the valuable feedback. Below we will address the concerns of the reviewer:

• Regarding: “However, the author overlooked DANet [1], a model that outperforms TabNet and provides a better interpretable framework. This could be included for comparison.”:

  We thank the reviewer for the valuable suggestion. We would like to point out that there exist 2 main differences between TabNet/IMN and DANet. TabNet/IMN provides local explanations, which are in the original feature space. While, on the other hand, DANet provides global explanations and it groups/abstracts higher-level features from the original feature space. This in turn makes DANet incompatible with our experimental protocol regarding interpretability.

  However, we believe DANet falls under methods that are explainable by design, and as such, we provide a comparison between all baselines and DANet in terms of performance in Figure 1 at the Global Response. As the reviewer suggested, DANet outperforms TabNet, however, it still performs worse compared to our proposed method in terms of accuracy.

  Our results are consistent with the results of previous work [1]. We will update the camera-ready version of our work to include DANet as an additional interpretable baseline.

[1] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.

We believe to have addressed all the concerns raised by the reviewer and we welcome any additional questions the reviewer might have.