Dear reviewer,
Thank you for your valuable feedback and insights. We answer your questions and comments below.

Weaknesses

In section 2, the encoding of numerical features involves the threshold $b_t$, and the thresholds are from the decision trees. So I wonder how we get these decision trees? Is the marginal effect of the model the same as those decision trees? And what about the performance compared with the decision trees?

Thank you for highlighting this point. Firstly, we acknowledge a typo in the PLE encoding. The correct form should be as follows:

$

z_{j(\text{num})}^{t} = \begin{cases} 0, & \ \text{if } x < b_{t-1} \\ 1, & \text{if } x \geq b_t \\ \frac{x - b_{t-1}}{b_{t} - b_{t-1}}, & \text{otherwise} \end{cases}
$

We have corrected it in the revised version of the manuscript.

In our approach, the encodings are computed in a target-aware manner. Specifically, as a preprocessing step, we fit a decision tree on pairs of the form $(x_j, y)$ for each feature $j = 1, ..., J$. The bins derived from the decision tree then define $b_t$. However, it’s important to note that the true value of $x$ is retained in the encoding, resulting $x \\mapsto z$, where $z \\in \\mathbb{R}^T$ is an enriched, higher-dimensional representation of $x$. Consequently, each feature subnetwork $f_j$ can yield different marginal predictions compared to the decision tree. Please also refer to [1] for a detailed introduction into embeddings for numerical features.

The method seems too trivial to achieve the expected performance. Only a linear layer and dropout can identify the marginal effects and even enhance the performance.

• Thank you for acknowledging the simplicity of our approach. A useful analogy is the mgcv::gam package in R, which transforms each feature into a higher-dimensional representation. In the case of GAMs, this is achieved through splines, whereas in our approach, we use PLE followed by the embedding layer. Splines are well-known for their ability to effectively capture marginal effects, as supported by sources [2, 3, 4]. As such, each feature network practically consists of an embedding layer, a possible activation function and an output layer. Since the input is already transformed to be in $z \\in \\mathbb{R}^T$, we find that these networks are capable of identifying marginal effects well as demonstrated in Table 1 and the provided figures.

In the tabular domain, it seems that only 8 datasets are used, which may raise doubts that these datasets were selected for this task.

• Thank you for this comment. We have based our selection on commonly used datasets in additive modeling. These datasets are also employed in studies such as [4, 5, 6, 7] and exceed the number used in those references. Nonetheless, as part of this revision, we have added additional datasets to the benchmark. So far, we have added 7 more datasets for 7 models, with ongoing efforts to expand both datasets and model comparisons. Preliminary results with these additional datasets align closely with the outcomes reported in our paper. Please see our general answer for the results.

It would be helpful to include a section on related work, such as transformers in tabular data, marginal feature effects in tabular or other domains, etc. This could help readers better understand the task and your method.

• Thank you for your valuable suggestion. We have included a section on related work, discussing transformers for tabular data and marginal feature effects, to provide additional context for readers. Currently, we have placed it in the appendix to prevent familiar readers from re-reading well-known concepts. If you feel it would be better suited in the main text or would benefit from additional elements, we are happy to revise accordingly.

It seems that in the Abstract, there are some spacing issues between lines.

Thank you for noticing this. We have adapted it in our revised version.

---

Thank you once again for your thoughtful feedback. We hope these responses provide clarification and reinforce our contributions.

---

[1] Gorishniy, Yury, et al. "On embeddings for numerical features in tabular deep learning." NeurIPS. 2022
[2] Rügamer, David, et al. "Semi-structured distributional regression." The American Statistician. 2024
[3] Wood, Simon N. "Thin plate regression splines." Journal of the Royal Statistical Society Series B. 2003
[4] Perperoglou, Aris, et al. "A review of spline function procedures in R." BMC medical research methodology. 2019

Dear Reviewer,

Thank you for your detailed feedback. We appreciate your time and suggestions.

From my perspective, the related work should be included in the main body of the paper, as this would enhance the completeness and readability of your paper.

• Thank you for your valuable suggestion. We will include the related work in the main body of the final version of the paper to enhance its completeness and readability.

Regarding the $b_t$, since you are fitting a decision tree for each feature, can I interpret this as leveraging the power of decision trees to drive the performance, rather than attributing the effect to your structural design? I would appreciate a clearer distinction between your method and decision trees.

• The PLE encoding, proposed by [1], is strictly a preprocessing method which can also be left out. Your interpretation is quite correct, in that we leverage it purely for performance reasons (see Table 2). We do not utilize predictions or any decision tree outputs other than the bin boundaries, $b\_t$. These boundaries can also be set continuously or based on data occurrence without relying on decision trees. For instance, methods like splines often use such continuous boundaries to great effect (see e.g. [2, 3, 4]).

• To further clarify, Table 1 illustrates that our method works effectively with alternative encodings, such as one-hot encoding or standardized input features, independent of decision trees. This versatility is further demonstrated in Table 2, where alternative encodings are also evaluated, showing that while decision-tree-derived boundaries offer a slight performance advantage, they are not a structural requirement of our method.

I am still skeptical that one MLP layer can achieve the performance claimed in the paper, and I will refer to the paper you recommend for a more comprehensive understanding.

• While we greatly value your skepticism in the review process, our claims are well-supported by extensive literature, experimental validation and our provided Repository. Single-layer architectures have demonstrated great performance, especially in Neural Additive Models (NAMs) and similar frameworks. For example, [4] shows that a single-layer, spline-based NAM outperforms more complex NAMs and Explainable Boosting Machines (EBMs). Similarly, [5] demonstrates the efficacy of shallow, parameter-efficient networks using low-rank polynomials.

• Furthermore, our study focuses on analyzing marginal effects without involving complex higher-order feature interactions, which aligns with the suitability of a shallow architecture. We encourage you to explore the provided codebase for additional empirical evidence. Given the strong support and reproducibility demonstrated in our approach, we hope this provides clarity and resolves your skepticism.

---

In the tabular domain, it seems that eight datasets are not sufficient. This may result in doubt that the datasets used are selected for this task.

• In our previous response, we addressed your concern regarding the number of datasets by nearly doubling the benchmarks, which required substantial effort. Since you had highlighted limited experiments as a core weakness, we kindly request that this significant improvement be acknowledged in your evaluation.

---

Thank you once again for your insightful feedback. We are committed to further improving our work based on your comments.

---

[1] Gorishniy, Yury, et al. "On embeddings for numerical features in tabular deep learning." NeurIPS. 2022.
[2] Wood, Simon N. "Generalized additive models: an introduction with R." (2017).
[3] Yeh, Raine, et al. "Fast automatic knot placement method for accurate B-spline curve fitting." Computer-aided design 128 (2020).
[4] Luber, M., et al. "Structural neural additive models: Enhanced interpretable machine learning." arXiv preprint (2023).
[5] Dubey, A., et al. "Scalable interpretability via polynomials." NeurIPS (2022).

Dear reviewer, we did our best to address your remaining concerns and answer your questions. In summary, we have:

• Detailed that decision trees are exclusively used during preprocessing and are only leveraged to determine the bin boundaries $b\_t$ and nothing else.
• Performed additional benchmarks to verify the superiority of NAMformer compared to other interpretable models.
• Addressed your remaining questions.

We look forward to hearing your feedback and continuing the discussion.

Dear reviewer,
Thank you for your feedback and valuable insights.

Given your comments and questions we were surprised by the particularly low score, especially as several points raised focus on well-established concepts and practices in the field. For example, adjustments to the placement of dataset descriptions (e.g., moving them from the appendix to the main text) are largely structural and do not impact the core contributions or rigor of our approach. We have provided detailed responses to each comment below, clarifying how our methods align with robust, widely accepted standards in the literature. We ask that these clarifications be fully considered in your overall assessment of the work.

The introduction and methodology sections could benefit significantly from writing improvements and more thorough explanation. In its current form, these sections assume a lot in terms of prior knowledge, e.g., of marginal feature effects, the FT-Transformer architecture, target-aware embeddings, and uncontextualized embeddings. These should be explained, at least at a high level.

• Thank you for this suggestion. Our work builds on fundamental concepts from additive modeling ([1, 2, 3, 4]) as well as tabular deep learning ([5, 6]). Following your recommendation, we have added a “Related Literature” section which is currently placed in the appendix to prevent familiar readers from re-reading well-known concepts. If you feel it would be better suited in the main text or would benefit from additional elements, we are happy to revise accordingly.

Additionally, from Section 2, it is somewhat unclear which parts are inherited from prior work on FT-Transformer and which parts are new. Particularly, I was confused about the feature encoding part--without referencing the prior work, I cannot easily assess if this part of the work is novel.

• We’re a bit puzzled by this comment. Feature encoding methods are well-established in tabular deep learning, and after double checking we do cite [5] accordingly to clarify that we do not introduce these encodings ourselves.

The authors should include specific use-cases in which the marginal feature effects are useful downstream. Otherwise, it is challenging to assess the significance of the contribution.

• The usefulness of marginal feature effects is well-established in both statistical modeling and interpretable machine learning ([1, 2, 3, 4, 7, 8]). Marginal feature effects provide direct insight into the relationship between features and model predictions, a cornerstone of interpretability that is critical in domains such as healthcare, finance, and policy-making. These are not niche or isolated use cases but foundational concepts widely recognized for their value in understanding and validating models. The significance of interpretability, particularly through marginal effects, is well-documented in the literature ([1, 2, 3, 4, 7, 8]) and should be evident to any audience, regardless of familiarity with specific methodologies. As such, we believe this comment reflects a misunderstanding of the fundamental role of interpretability in applied machine learning.

• We have added a short paragraph in the "Realted Literature" section in the appendix and could incorporate this into the main part. However, explaining each and every concept of well established themes is beyond a conference submission and boarders on a general and basic introducion into tabular modeling.

The paper should have a standalone explanation of the datasets that are being used in evaluation beyond saying that the evaluation is done on four regression and four binary classification datasets—what are these datasets and where do they come from?

• We are somewhat surprised by this comment, as the datasets and their sources are described in detail in the appendix, following the standard structure of studies in this field ([1, 2, 3, 4]). Given the page limitations and the general consensus in the field that benchmark dataset descriptions are typically included in the appendix [1–6], we are concerned that moving these descriptions to the main text would compromise the quality of our work. However, we are open to further discussion if you feel strongly otherwise.

Furthermore, the evaluation should be more thorough than regression and binary classification -- conceivably, the proposed method could become worse than FT-Transformer as the number of classes grows. The authors should include such tasks.

• Please refer to our general answer and our revised manuscript where we have included additional benchmarks. We have oriented on the literature, especially for interpretable modeling, i.e. [1-4, 7-8] and have thus focused on regression and binary classification tasks. It is worth noting, that 90% of datasets in the tabular domain on openML are regression or binary classification task datasets. Since our theoretical justification also holds for multi-class classification (See appendix C) and the number of classes have no effect on the architecture other than the number of output nodes, we disagree that there is any theoretical reasoning why NAMformer should perform worse than FTTransformer for multi-class classification problems.
  To further strengthen this, we have performed 10-fold cross validation on the red and white wine quality datasets taken from UCI ML Repo and compared NAMformer to FTTransformer with identical hyperparameters. The results show that, as expected, there is no difference for multi-class classification and NAMformer performs just as good as FTTransformer for multi-class classification tasks.

Questions

What are uncontextualized embeddings

• Uncontextualized embeddings are fixed vector representations of words/inputs/tokens, generated independently of the surrounding context. That is, the embeddings before being passed through any joint layers.

What does token identifiability mean

• Token identifiability refers to the ability to distinguish a specific token (e.g., a word/input) within a sequence/features based on its unique representation. It ensures that the model can differentiate tokens, even in similar or repeated contexts, to preserve meaning and structure. Please refer to [9] for a detailed description.

Both, token identifiability and (un)contextualized embeddings are well known concepts in ML, which is why we did not include specific definitions in our paper.

---

Thank you once again for your feedback. We hope these responses provide clarification and reinforce our contributions.

---

[1] Agarwal, Rishabh, et al. "Neural additive models: Interpretable machine learning with neural nets." NeurIPS 2021
[2] Chang, Chun-Hao,et al. "NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning." ICLR 2022.
[3] Thielmann, Anton, et al. "Interpretable Additive Tabular Transformer Networks." TMLR 2024.
[4] Radenovic, Filip, et al. "Neural basis models for interpretability." NeurIPS 2022.
[5] Gorishniy, Yury, et al. "On embeddings for numerical features in tabular deep learning." NeurIPS 2022.
[6] Gorishniy, Yury, et al. "Revisiting deep learning models for tabular data." NeurIPS 2021.
[7] Hastie, Trevor J. "Generalized additive models." Statistical models in S." Routledge, 2017.
[8] Hastie, Trevor, and Robert Tibshirani. "Generalized additive models: some applications." JASA 1987. [9] Brunner, Gino, et al. "On identifiability in transformers." arXiv preprint arXiv:1908.04211 (2019).

Dear reviewer,
Thank you for your valuable feedback and insights.

The definition given for at line number 279 doesn't make sense to me - it could even take the opposite sign of the overall risk $R\_{\\tilde{\\mathbf{w}}\_{-k}}$ if the model has a greater loss with just feature than on average.

• We fully agree that the describing $R\_{\\tilde{\\mathbf{w}}\_{-k}}$ as the "risk not associated with $\\tilde{\\mathbf{w}}\_{-k}$" is misleading. We think that the term "difference between the overall risk and the risk associated with $\\tilde{\\mathbf{w}}\_{-k}$" would be more appropriate. We have revised the manuscript accordingly. We also agree that this quantity is negative if the model has a greater loss with just feature $k$ than on average, which will typically be the case.

This further makes sense in the context of equation (5) as the risk regarding the marginal effect $\\mathbb{E}\_{x\_k, y \\sim P^{\\mathcal{D}}} \\left [\\mathcal{L} \\left ( \\beta\_0 + f\_k(x\_k) , y\\right)\\right ]$ is decreased by a negative quantity $R\_{\\tilde{\\mathbf{w}}\_{-k}}(1-p(\\tilde{\\mathbf{w}}\_k))$ compared to the overall risk $R$. I.e., $- R\_{\\tilde{\\mathbf{w}}\_{-k}}(1-p(\\tilde{\\mathbf{w}}\_k))$ increases the risk of identifying the marginal effect.

$\\mathbb{E}\_{x\_k, y \\sim P^{\\mathcal{D}}} \\left [\\mathcal{L} \\left ( \\beta\_0 + f\_k(x\_k) , y\\right)\\right ] = \\frac{R - R\_{\\tilde{\\mathbf{w}}\_{-k}}(1-p(\\tilde{\\mathbf{w}}\_k))}{p(\\tilde{\\mathbf{w}}\_k)} \\ \\ (5).$

Equation (7) depends on the assumption that the risk for each single-feature dropout vector is the same and equal to $R\_{\\tilde{\\mathbf{w}}\_{-k}} = R \\cdot (1-p(\\tilde{\\mathbf{w}}\_k))$, which seems very unlikely for realistic models or datasets where features have different importances and the model tends to perform better given more features. So I don't see the relevance of this result.

• We included this result because it provides an intuitive upper bound on the risk for identifying a specific marginal effect $\\mathbb{E}\_{y|x\_k}{\\left [ y | x\_k \\right ] }$. We agree that the assumption of equal risk for each single-feature dropout vector is strong and may not hold in practice. However, we think that this result is still useful for understanding the behavior of the model and the dropout procedure. We now clearly point out in the manuscript that this case is unrealistic and that the result is a theoretical upper bound.

Could you please clarify the results in Section 2.1, especially how to interpret them with respect to interpretability?

• Our goal of section 2.1 is to relate the risk in identifying a marginal effect, i.e. $\\mathbb{E}\_{x\_k} \\left [ \\mathcal{L}\\left ( \\beta\_0 + f\_k(x\_k), \\mathbb{E}\_{y|x\_k}{\\left [ y | x\_k \\right ] } \\right ) \\right ]$ to the overall risk $R$ when performing feature dropout in order to provide theoretical insights. The finding in equation (6) can be summarized as having proven that minimizing the ratio between the difference between overall risk and the risk associated with $\\tilde{\\mathbf{w}}\_{-k}$ and, second, the dropout probability for only keeping the $k$-th vector implies a low risk in identifying marginal effects. We added this explanation to the manuscript to clarify the interpretation of the results.

• While stricter theoretical bounds would certainly be ideal, even though perhaps not even possible, we believe that our results and framework still provide valuable insights into the behavior of feature dropout. Especially because, to our very best knowledge, we are first to provide such a theoretical analysis.

• We would also like to point out that we provide additional comprehensive empirical results in section 3 that show that NAMFormer outperforms other commonly used methods in terms of identifiability of marginal effects---even GAMs and linear models. It is thus very likely that, for instance the fact that the NAM-Part of our methodology can fit the marginal effects of the model very well, is a key factor in the success of our method which we do not capture in the theoretical analysis.

Dear reviewer,

Thank you for your detailed follow-up and for carefully reviewing the updates we have provided.

The added literature review is useful - I would definitely suggest adding at least some of it to the main body so that readers are more aware of the context of the work.

• Thank you for this advice. We will await the other reviewers comments before adapting these changes, but we will definitely include a broader literature review in our final version.

I still would have preferred to see an explicit comparison of the proposed method to post-hoc feature attribution methods like integrated gradients, because I think a natural question when looking at this work is "why not just train an ordinary FT-Transformer and estimate feature effects afterwards?".

• Thank you for this question. While Integrated Gradients (IG) [1] and other post-hoc methods are powerful for detecting feature importance, particularly in classification tasks, they may face challenges in identifying global marginal effects, especially for regression tasks. Some of these challenges and limitations are discussed in [2]. For IG in particular.

• Baseline Sensitivity and Path Dependence:
  IG computes attributions by integrating gradients along a path from a baseline $\\mathbf{x}\_{\\text{baseline}}$ to the input $\\mathbf{x}$: $\\text{IG}\_i(\\mathbf{x}) = (x\_i - x\_{i, \\text{baseline}}) \\int\_{\\alpha=0}^1 \\frac{\\partial f(\\mathbf{x}')}{\\partial x\_i} \\Big|\_{\\mathbf{x}' = \\mathbf{x}\_{\\text{baseline}} + \\alpha(\\mathbf{x} - \\mathbf{x}\_{\\text{baseline}})} d\\alpha.$ The choice of baseline affects how the path traverses the input space, influencing attributions significantly, especially for features involved in interactions. For instance, the attribution of a feature $x\_i$ in the presence of an interaction term $f\_{12}(x\_i, x\_j)$ depends on the values of $x\_j$ along the path. A poorly chosen baseline may exaggerate or understate the contribution of interactions, leading to attributions that are inconsistent with the actual marginal effect.

• To illustrate this, we have added a Jupyter Notebook in our Github repository. The notebook demonstrates that IG fails to accurately identify marginal effects, even in cases of simple additive data-generating processes without interaction effects.

Additionally, we included an FT-Transformer in our ablation study and extracted marginal effects using Integrated Gradients, as per your suggestion. The results, shown below, demonstrate that IG struggled to accurately identify marginal effects in regression problems with feature interactions, consistently performing poorly and even yielding negative correlations across the board. While these results at a first glance might be surprising, the results in our notebook (plots are at the bottom) demonstrate the methods struggles, even for datasets without any interactions.

Dear Reviewer,

Thank you for your thoughtful feedback and continued support. We’re happy to hear that your concerns have been addressed. We will move the related work section into the main body in the final version as suggested. We also plan to expand the discussion and limitations section to better incorporate your comments, building on the points already addressed.

---

Thank you again for your insights and for raising your score. We’re committed to making further improvements and ensuring the manuscript reflects your helpful suggestions.