Interpretable Generalized Additive Models for Datasets with Missing Values

Thank you for your review! We appreciate your recognition that this work is well written, novel, and contains extensive experiments, and we are open to discussing any additional concerns you might have.

The proposed method is constrained by its reliance on l_0 regularization.

While $\ell_0$ regularization has historically been difficult to optimize, a key strength of our paper is that it leverages the fast optimization framework from Liu et al. 2022, which introduces several computational tricks that make it quite manageable. This lets us gain the substantial benefits of $\ell_0$ regularization (extreme sparsity in a setting that is prone to producing dense models) without suffering substantial runtime costs.

Thank you for the thoughtful review – we appreciate your recognition that sparsity and interpretability are particularly important for practitioners. We hope we have addressed each of your concerns below, and look forward to any ongoing discussion.

The title should convey that the study is limited to GAMs

This is a fair point – we plan to update the title to something like “Interpretable Generalized Additive Models for Datasets with Missing Values”. GAMs are currently among the most popular forms of interpretable ML models [1].

[1] https://github.com/interpretml/interpret

Runtime comparison in 4.3 is a bit vacuous because it depends on the exact implementation, etc. Maybe looking at computational complexity would be more interesting.

While computational complexity would provide an interesting additional kind of comparison, we believe practical runtime is the more relevant metric here because it is more reflective of what users will experience. This is particularly true for methods like FastSparse, where computational tricks substantially improve the algorithm’s practical runtime without necessarily affecting its big O runtime. While runtime can vary with implementation, we deliberately used implementations that are well established in prior work whenever possible. As such, the results in 4.3 should be fairly reflective of what an actual user would experience, as they would likely use these same implementations.

Empirical evaluation is somewhat limited

We are unsure in what way this is meant, but note that appendix E contains a variety of additional empirical results, including evaluation over six datasets using a wide variety of baseline methods. Together, sections D and E of the appendix are longer than the entire main paper and consist of empirical evaluation.

l.27, it seems strange to contrast the choice of (not) using missingness with the choice of using l0 regularization. They seems to be rather orthogonal choices.

Yes, they are orthogonal choices. We think this is just ambiguous phrasing – we meant to contrast handling missingness alongside $\ell_0$ regularization with handling missingness without $\ell_0$ regularization. We will update the wording to make this clearer.

Why l0 and not some other regularization?

$\ell_0$ regularization is beneficial because it produces very sparse models, and this paper is focused on interpretability. In the context of M-GAMs, this means fewer variables will contribute to model predictions and therefore fewer shape functions will need to be presented. This makes it easier for users to interpret model predictions.

It seems questionable to encode reasons for missingness with natural numbers, as these imply an ordering.

Good point, we’ll change it to letters (or another notation you suggest that implies no ordering). This does not actually affect the implementation since M-GAM is only exposed to the binary indicator variable for each missingness reason. If this is referring to the FICO description from the dataset description, it was not our choice to use the stated missingness encodings; those are from the creators of the dataset.

It looks like all other methods have better mean accuracy than M-GAM in Figure 6, Breast Cancer dataset. Is this correct? If so, the text should discuss this rather than glossing over it.

This is true, but reflects an oversight in creating Figure 6. Breast Cancer and MIMIC are both heavily imbalanced, and as such performance is better measured via AUC, which is done in other plots. Accuracy is not reflective of quality for imbalanced datasets. Under this metric, it is not true that the median AUC for M-GAM is worse than all other methods, as shown in the combined response. The class balance for each of the four datasets is as follows

Breast Cancer: 78.2% negative

MIMIC: 89.6% negative

Pharyngitis: 54.9% negative

FICO: 47.8% negative

The authors state interpretability as one of M-GAM's main advantages. I would like to see this made explicit, e.g., with an example, especially in comparison to existing methods.

We intended for Figures 1 and 2 to serve as such examples, but believe that additional examples on the datasets we study might help prove this point. We will prepare such examples, and add them to the appendix.

We appreciate your review, and your recognition of this work’s novelty, theoretical foundations, and clear writing. We have responded to each of your criticisms below, and look forward to any continued discussion.

My understanding is that the setting considered in this paper seems to be limited to GAMs with main effects and not higher-order GAMs (in particular, GAM with pairwise interaction effects, GA2M [Lou et al., KDD 2013]).

This is generally correct, although M-GAMs could naturally be extended to include interaction effects of an arbitrary order – both between missingness and between observed values, and between different combinations of observed values. However, we note that, even in Lou et al. 2013, higher order GAMs showed substantially improved classification performance relative to main effect GAMs only on digit recognition tasks (“Letter”, “Gisette”) and not on datasets that would be treated as tabular today. We did experiment with additional interaction terms, along the lines of GA2M, but did not find any scenarios where this improved performance. As such, we did not feel comfortable including it in the paper.

There has been significant work on handling with missing values in decision trees (that are also a widely used interpretable model). There is very little both in terms of discussion (e.g., are the approaches there applicable to GAMs, does the proposed approach share some similarities to existing approaches in decision trees) as well as in terms of experimental results (how do decision trees with missing values compare to GAM with missing values, at the moment only SMIM with decision tree is considered and not approaches that are inherent to decision trees).

This is a fair point. We will add discussion of this work in the related work section, and have added comparisons to some decision tree based methods that explicitly handle missingness as implemented in SKLearn, as shown in the shared response. In general, because these papers introduce changes that are specific to decision trees, such as default traversal paths when data is missing, they are not directly applicable to M-GAM. We found that decision trees with this explicit handling improved runtime and offered interpretability, but generally had poorer performance than M-GAM. Random forests offered comparable performance and runtime, at the cost of interpretability.

In summary, we’re happy to address this work in related work, but it does not decrease the value of our own approach, which performs well and is separate from the decision-tree-inherent methods you’ve mentioned.

The experimental results on real datasets do not demonstrate strong improvement in performance (but there is improvement in other factors, e.g., runtime compared to impute-then-predict).

While this is true, the fact that M-GAM has comparable performance to other methods on real data means that our runtime and interpretability gains come with no substantial costs. M-GAM quickly provides a sparse, transparent model that is as accurate as complex impute-then-predict baselines, which we see as a strength.

The analysis on other models, including decision trees (with and without SMIM) does not include runtimes so it is not clear if the new model is also significantly faster from, say, decision trees or logistic regression or random forests with SMIM.

We have added some such comparisons in the global response. Please note that the SMIM framework consists of both imputing missing data and providing missingness indicators, meaning that it is no faster than the given runtimes with imputation. Additionally, these methods that use imputation introduce complexity, sacrificing interpretability in a way that M-GAM avoids.

I would appreciate the authors response to the weaknesses listed above. In addition I was wondering if the authors have considered incorporating this mechanism into one of the recent neural GAM approaches, e.g., Neural Additive Models [] which will allow potentially for better scalability?

We hadn’t previously, but it's an interesting idea. However, based on the results presented in [1], it seems that NAM’s will not actually scale better than M-GAMs. Our current $\ell_0$ regularized method is already quite fast. If we were to discard our approach in favor of a Neural GAM with some other sparsity approach, we don’t anticipate benefits to scalability because such an approach would effectively need to fit neural nets for up to (# features)$^2$ shape functions to cover missingness interactions, and a sparsity-regularized neural GAM can take quite a while even for a simple dataset. For example, table 5 of [1] shows a neural GAM taking ~30 seconds on a 6172 sample, 13-feature dataset, COMPAS, which is slower than our method on the much more computationally difficult FICO dataset (10,459 samples, 23 features), even when we augment the FICO dataset with missingness interactions.

[1] Shiyun Xu, Zhiqi Bu, Pratik Chaudhari, Ian J. Barnett. Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity. ECML PKDD 2023

Thank you for your review. We appreciate your recognition of the novelty of this work and its strong experimental backing. We look forward to discussing any ongoing questions or concerns you may have.

I think the main weakness of M-GAM is its performance which just barely match other baselines' performance. Since it is trained end-to-end manner, we can expect more performance gain than impute-then regress methods since it directly uses supervision during training. Since informative missingness is rare in real world dataset, predictive performance of M-GAM on real world dataset can harm the applicability.

The goal of this work is to improve interpretability without harming performance, and we accomplished that. M-GAM is consistently among the most performant models on each dataset. Sometimes, it even improves both performance and interpretability. By producing an interpretable model, we allow practitioners to easily identify confounded reasoning and use the model more responsibly. This ability to troubleshoot helps improve overall accuracy of the system during and after deployment, not just performance on a static dataset.

why M-GAM especially good on MAR setup?

M-GAM is particularly effective in the MAR setup used in the semi-synthetic example because the value is MAR with respect to the label. That is, in that semi-synthetic case, there is some signal about the outcome embedded in whether or not a feature is missing.

This can happen in a variety of settings; for example, imagine conducting a survey to help predict whether individuals care about privacy or not. If we ask for a home address and an individual doesn’t answer the question, resulting in missing data, that might be a strong indicator that they do care about privacy.

Thank you for your review. We hope we have addressed your primary concern below, and look forward to any further discussion.

Proof only works since the unobserved noise for k1 and k2 is chosen as it is... If the assumption is switched or equal signs added the proposition will fall apart. It seems the proposition needs to be far more limited than it is right now.

In 108 should it than not say "Corollary 3.2 states that perfectly imputing missing data can reduce the best possible performance of a predictive model, if the noise of the assumed predictor dependent missingness mechanism is lower than the measurement noise."?...

Proposition 3.1 and Corollary 3.2 are existence claims, which we prove by construction. It is valid and standard to prove such claims by constructing a single viable Data Generating Process (DGP); such a proof does not imply that DGP is the only case where our claim holds. In fact, in situations like the one you've named, where missingness information is higher noise than the rest of the data, there still exist cases where missingness information is needed to achieve the best possible model - we just chose not to focus on them to keep the proof simple. As an example, we have constructed a case where the noise in the missingness indicator is the greatest (k_1<...<k_2<0.5), but the Bayes’ optimal model with missing data is still superior to that with perfect imputation:

Consider a case similar to that used for the proof in proposition 3.1, where we add an additional variable $X_3$ and, as requested, have the noise for the missingness be higher than any other noise.

$Y = |X_1X_2 - \epsilon_1|, \epsilon_1 \sim \textrm{Bern}(\frac{1}{6})$ $M = |Y - \epsilon_2|, \epsilon_2 \sim \textrm{Bern}(\frac{1}{4})$ $X_3 = |Y - \epsilon_3|, \epsilon_3 \sim \text{Bern}(\frac{1}{5})$

We also adjust the probabilities for $X_1$ and $X_2$ being true so that this is a balanced classification problem: $X_1, X_2 \sim \textrm{Bern}(\frac{1}{\sqrt{2}})$, so $X_1X_2 \sim \textrm{Bern}(\frac{1}{2})$

The complete proof for this case is available at the bottom of this rebuttal.

Why did the authors impute the data for RF? If C4.5 is used than the missingness can be handled without explicit imputation by change of the impurity function. There are also internal impute proposals: Stekhoven & Bühlmann (2012), "MissForest—non-parametric missing value imputation for mixed-type data", Bioinformatics, 28, 1.

Avoiding explicit imputation is a valid alternative baseline, which we have added, as shown in Figure 1 of our shared response. The results of this experiment align with those for previous baselines: random forests and decision trees without imputation do not outperform M-GAM. Additionally, we note that MissForest is used as an imputation method in Section E.2 of the supplement. We find that MissForest imputation (including MissForest used with a random forest model) does not outperform any of our other impute-then-predict random forest baselines.

Weaknesses:

I would have run 2 distinct experiments at least. One with synthetic missingness where the ground truth is known and the missigness can be increased. Additionally I would have liked to see also a mix of MCAR and MAR to demonstrate the results when the assumed mechanism is incorrectly chosen.

We don’t actually make any strict assumptions about the missingness mechanism. We regularize towards MCAR through our $\ell_0$ regularization, but this is not a strict assumption (for example, we still handle the MAR setting in figure 3). As per your suggestion to include MCAR missingness, we’ve included a plot where we switch to MCAR instead of MAR missingness for the experiment done in figure 3 (plot included in the general response). We still handle missingness as well as our competitors, though there is no informative missingness that we can take advantage of in this setting.

The authors emphasize the interpretability enabled by M-GAM and figures are shown. It would have been nice to discuss the interpretation in the text.

We agree that additional discussion of how to interpret an M-GAM would be appropriate – we will add more discussion around Figure 2, and several more visualizations will be added to the supplement.

Proof:

The bayes optimal model with perfect imputation of $X_1$, and no access to $M$, is still just to predict in accordance with $X_1X_2$ for $P(X_1X_2=Y)=\frac{5}{6}$. When $X_1X_2=X_3$, all information available suggests $X_1X_2$ is correct. When $X_1X_2\neq X_3$, we still have the bayes optimal prediction aligning with $X_1X_2$: $P(Y=1|X_1X_2=1,X_3=0) =\frac{5}{9}>0.5$ and $P(Y=0|X_1X_2=0,X_3=1)=\frac{5}{9}>0.5$.

If we instead only have access to $X_1$ when it is not missing, but we also know when $X_1$ is missing (i.e. we know $M$), then the following approach will perform better than the above model: