Correcting misinterpretations of additive models

Significant Weakness 1:

We appreciate the reviewer’s thoughtful suggestion. We agree that a more explicit theoretical argument and working definition for suppressor variables would strengthen the paper.

In the revised version, we will:

1. Provide a clear working definition of suppressor variables in the context of additive models, grounded in prior literature (e.g., Conger, 1974; Haufe et al., 2014; Wilming et al., 2023). In our context, we define a suppressor as a variable that improves predictive performance by carrying information about other features, but has no statistical dependency with the target variable. This contrasts with informative variables, which are directly associated with the label.

2. Formally demonstrate that PatternGAM avoids attributing importance to such suppressor variables, based on the Statistical Association Property (SAP) introduced by Wilming et al., (2023). Specifically, we will show that the pattern value - which reduces to the covariance between the nonlinearly transformed feature and the target - is zero when no statistical dependency exists. This allows PatternGAM to disambiguate suppressors and noise from confounders and informative features.

3. Expand our intuitive explanation with a more general argument: when the NAM learns a linearly separable representation from potentially nonlinear input space, PatternGAM projects the learned decision boundary back through the data distribution.

Conger, A. J. (1974). A Revised Definition for Suppressor Variables: a Guide To Their Identification and Interpretation. Educational and Psychological Measurement, 34(1), 35-46

Haufe, Stefan, et al. "On the interpretation of weight vectors of linear models in multivariate neuroimaging." Neuroimage 87 (2014): 96-110.

Wilming, Rick, et al. "Theoretical behavior of XAI methods in the presence of suppressor variables." International Conference on Machine Learning. PMLR, 2023.

Significant Weakness 2:

PatternGAM is designed to explain additive models such as NAMs or EBMs. In the XOR scenarios, for example, the NAM architecture itself exhibits limited predictive performance compared to the MLP (see Figure 5), reflecting a mismatch between model capacity and data complexity. SHAP, PatternAttribution, and Integrated Gradients are applied to the MLP that perform better on these tasks. Even with near perfect classification performance, methods like SHAP and Integrated Gradients have been shown analytically to assign importance to suppressor features (Wilming et al., 2023), and PatternAttribution has shown suboptimal performance in non-linear tasks where suppressors are present (Clark et al., 2024).

Thus, while attribution methods on MLPs may appear to outperform PatternGAM when evaluated the FNI-EMD metric, this is in part a result of comparing explanations across fundamentally different model types. When restricted to additive model architectures, PatternGAM offers significant advantages in not attributing false positive importance to suppressors, as we demonstrate in both synthetic and real-world settings.

Minor weakness 1:

Thank you for pointing this out. We agree that the current version of Figure 1 may be unclear without the context provided in the appendix, particularly regarding the meaning of the orange boxes. To clarify:

1. The orange boxes indicate distractor regions - i.e., suppressor features that carry no statistical association with the target but influence the model’s decision through correlations with informative features.

2. We will update the figure caption and lines 230–234 to explicitly define the orange regions and distinguish them from noise or overlapping regions. Additionally, we will clarify that PatternGAM suppresses not only distractors (suppressors), but also any features with no statistical association to the target.

Minor weakness 2

Thank you for raising this point. Our intent was not to overstate this capability but to highlight an interesting outcome in the COMPAS task: PatternGAM neutralises the attribution to race and sex, suggesting that, in this model, these features behave more like suppressors than confounders or true predictors of recidivism. We will revise the text in Section 4.2 to more clearly distinguish between suppressors and irrelevant features, which PatternGAM is explicitly designed to identify and suppress; versus confounders, which may still receive attribution if they are statistically associated with the label.

Question 1:

We can see in Figures 3 and 4 that the many lighter shaded lines are individual models from the ensemble, showing the variation and stochastic nature of the NAM optimisation process. The emboldened line can be seen as the ‘mean function’, or the reviewer’s idea of a weighted feature-wise combination. Taking an ensemble of many NAMs allows us to capture this average effect and account for such variation.

Question 2

Thank you for raising this interesting observation. First, we would like to clarify that PatternGAM is a post-hoc interpretability method, not a model architecture. It does not affect model performance - it interprets the predictions of an already trained additive model (NAM or EBM), including when interaction terms are present. We will revise the terminology in the final version to better reflect this distinction.

In both COMPAS and MIMIC-IV, we observed that adding pairwise interaction terms yielded only minor improvements in AUROC (e.g., 0.821 without interactions and 0.824 with interactions for the MIMIC-IV setting). This suggests that the majority of predictive signal in these tasks is already captured by univariate nonlinear effects, and the added value of interactions is limited due to redundancy or diminishing returns. For instance, in MIMIC-IV, many features (e.g., age, prior admissions, lab values) are already strong individual predictors.

Question 3

As discussed in Haufe et al. (2014), sparsity-inducing regularisation does not eliminate suppressors; in fact, suppressors can persist in even the sparsest optimal models if they help cancel noise in correlated predictors. PatternGAM’s goal is not to remove suppressors from the model, but to avoid assigning significant importance in explanatory importance. While LASSO could produce more compact explanations, it would not distinguish suppressors from true predictors, and may bias the pattern attribution undesirably.

Question 4

While sparse GAM methods such as FastSparse aim to improve interpretability by concentrating predictive power into a minimal subset of features, they do not eliminate suppressor variables. As shown in Haufe et al. (2014), suppressors can remain essential for optimal predictive performance and may be retained even under heavy sparsity constraints. PatternGAM addresses this and will not attribute importance to suppressors.

In that sense, PatternGAM complements sparse GAMs: it can be applied to any trained additive model, including sparsified ones, to produce corrected attributions that distinguish true signal from suppressors.

Question 5:

Regarding Equation (7): the matrices $\gamma_{uv}$​ and $C_{uv}$​ do not refer to interactions between pixels $u$ and $v$. Instead, they define the mass flow and cost of transporting explanation values between individual pixels in the FNI-EMD metric. In this formulation, the ground truth consists of individual important pixels, and the transport cost is set to zero between any pair of ground-truth pixels, regardless of spatial distance. This allows the metric to ignore false negatives, as intended.

Regarding the ground cost matrix C′ for interactions: you're correct that the current draft does not specify this in full detail, which we will update. For interaction terms (e.g., $Z_{ij}$​), the ground distance is defined as the sum of Euclidean distances between both constituent pixels.

Also, does FNI-EMD prefer subset-style explanations over complete ones, or does it merely refrain from additionally penalizing subsets?

FNI-EMD does not prefer subset-style explanations over complete ones - it simply refrains from penalising explanations that are subsets of the ground truth. FNI-EMD transforms any explanation into the closest subset of the ground-truth support (effectively projecting the explanation onto the ground-truth region), and then computes the transport cost needed to move the attribution mass to that subset. This projection incurs zero cost, so any explanation that is a subset of the ground truth - whether partial or complete - receives the same (perfect) FNI-EMD score.

Bonus question out of curiosity: On the synthetic vision problem, can the results obtained by this method be compared to an interpretable case-based reasoning neural network, such as ProtoPNet?

Thank you for the interesting question. We believe that PatternGAM and ProtoPNet serve fundamentally different explanatory goals. PatternGAM provides feature-level attributions, identifying which parts of the input contribute to the model’s decision, with respect to the underlying data distribution. In contrast, ProtoPNet offers a case-based reasoning approach, explaining predictions by referencing similar training examples or learned prototypes.

Weaknesses

1. The operation of premultiplying the weights by the covariance matrix requires a formal proof to guarantee its validity.

This is straightforward to show. In the revised paper, we will explicitly define the Statistical Association Property (SAP), as formalised by Wilming et al. (2023). This property ensures that features with zero statistical dependency on the label (even nonlinearly) receive zero attribution. In the revised paper, we will provide a formal proof that the covariance adjusted shape functions and weights have the SAP property. We will show this for main effect (non-interaction) features, however extending the SAP to interaction features is non-trivial and will be an interesting direction for future work.

Wilming, Rick, et al. "Theoretical behavior of XAI methods in the presence of suppressor variables." International Conference on Machine Learning. PMLR, 2023.

2. The computational complexity increases explosively with dimensionality, rendering the method hard to use in practice; the paper should explore approximate strategies that can alleviate this issue.

We appreciate the reviewer’s concern regarding scalability. While the presentation of PatternGAM using covariance-based correction may give the impression of high computational complexity, the method is efficient in practice. Specifically, PatternGAM reduces to computing the covariance between each (individual) nonlinearly transformed feature (or interaction term) and the target, which can be done without explicitly forming the full covariance matrix or computing matrix-vector products. We will include a formal derivation of this in the revised version.

3. The empirical evaluation omits comparisons with several mainstream explanation techniques—such as SHAP, LIME, and Integrated Gradients—whose inclusion is necessary for a fair assessment.

We thank the reviewer for the feedback. However, we would like to clarify that both SHAP and Integrated Gradients are included in our empirical evaluation. These methods are reported in the final two columns of Table 1 (as well as Tables 2 and 3 in the appendix), and correspond to columns 7 and 8 in Figure 6.

While we did not include LIME in our evaluation, this was a deliberate choice. Prior work (e.g., Wilming et al., 2022, 2023; Clark et al., 2024) has highlighted that LIME often fails to recover informative features in synthetic benchmarks with structured ground truth, such as ours. We therefore prioritised attribution methods with stronger theoretical foundations or empirical performance in comparable settings. We will clarify this decision and cite supporting work in the revised manuscript.

Clark, Benedict, Rick Wilming, and Stefan Haufe. "XAI-TRIS: non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance." Machine Learning 113.9 (2024): 6871-6910.

4. Purely additive model structures are uncommon in real-world applications, which inherently restricts the practical reach of the proposed approach.

We appreciate the reviewer’s point that purely additive model structures are less common in unconstrained real-world modeling. However, additive models are highly relevant and widely used in recent practice ,particularly in domains that require interpretable or auditable decision-making. For example, Caruana et al. (2015) showed that an intelligible GAM with pairwise interactions could match black-box model accuracy in predicting pneumonia risk while surfacing and correcting clinically implausible patterns. Ravindra et al., (2019) studied multiple applications of GAMs to link air pollution and climatic variability with adverse health outcomes. Yang et al. (2025) recently showed that GAMs can be effective for predicting energy savings based on measurement and verification of efficiency measures in commercial buildings, which is a key requirement of sustainability goals. In such contexts, avoiding false positive attributions to suppressor variables is essential, making PatternGAM’s contribution directly relevant to real-world deployment of interpretable models.

In our paper, we demonstrate the applicability of PatternGAM on two real-world datasets: COMPAS, a widely used fairness benchmark involving criminal recidivism prediction, and MIMIC-IV, a large-scale clinical dataset representative of real-world ICU decision-making. These reflect genuine use cases where interpretability is critical, and where additive models like NAMs and EBMs are increasingly adopted as presumed "glassbox" alternatives to black-box predictors. in the Discussion section of the revised manuscript (around lines 323-342), we will make it more explicit that there has been a recent increased interest in tabular data and tabular foundation models, and link this to existing writing on the popularity of using additive models for tabular data.

Caruana, Rich, et al. "Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission." Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015.

Ravindra, Khaiwal, et al. "Generalized additive models: Building evidence of air pollution, climate change and human health." Environment international 132 (2019): 104987.

Yang, Jian, et al. "Climate adaptive energy efficiency modeling using a generalized additive approach to optimize building performance across Chinese climate zones." Scientific Reports 15.1 (2025): 20088.

5. The method cannot remove dependencies stemming from nonlinear correlations; although the authors acknowledge this limitation, it remains important to analyse how influential such dependencies become in more complex settings.

We thank the reviewer for raising this point. However, we would like to clarify that PatternGAM is explicitly designed to detect and attribute non-linear dependencies between features and the target, as long as these are captured within the model structure -such as univariate nonlinearities or explicit interaction terms. This aligns with the assumptions of the underlying GAM or NAM model: if the model captures a given nonlinear dependency, PatternGAM will reflect it in the attribution. The method is not restricted to linear correlations. In fact, in our evaluations (e.g., on the XAI-TRIS datasets, which include structured noise and nonlinear distractors), PatternGAM performs robustly even in the presence of nonlinear correlations, outperforming other methods in identifying true feature relevance. We also include the non-linear dependence measures Mutual Information and the Hilbert-Schmidt Independence Criterion (HSIC) in Table 4 in the appendix to show that dependencies captured are not merely linear. To clarify, we do not claim in the paper that PatternGAM is limited to linear correlations, nor does the method filter them out - it highlights statistical associations as learned by the model, including nonlinear ones. If a GAM is not expressive enough to capture complex dependencies (e.g., beyond included pairwise or higher-order terms), then both model performance and explanation quality may degrade. In such cases, we recommend that explanations only be interpreted for well-performing models, where the learned representation is likely faithful.

Thank you for the accurate and thoughtful summary of our work. We have had to remove a lot of explanation of our responses and have combined responses on repeated points/questions (for example, generality of PatternGAM, a local version of PatternGAM) due to rebuttal length constraints. Please feel free to ask for further clarification over the next week!

The method is tailored to additive model families (GAMs, NAMs, EBMs) and cannot directly be applied to more expressive or end-to-end deep models (e.g., CNNs, transformers).

We see PatternGAM as a first step toward bringing the soundness of the activation pattern framework to non-linear architectures. We chose (neural) additive models as a target because they are actively promoted as “intrinsically interpretable” models - yet, as we show, they are still vulnerable to suppressor-induced misinterpretations. PatternGAM addresses this by correcting how importance is attributed post-hoc, without requiring inverse mapping assumptions. Generalising PatternGAM to end-to-end deep models like CNNs and transformer architectures would require a principled way to extract structured, linearly separable representations suitable for pattern-based transformations - a challenge we see as promising future work.

While PatternGAM adjusts explanation attribution, it doesn’t alter model training or prediction. The approach could miss interpretability issues that arise during optimization or from model biases.

We agree that PatternGAM, by design, does not alter model training or predictions - it operates entirely post-hoc to analyse learned representations. While it's unclear which specific interpretability issues the reviewer refers to, we would like to emphasise that PatternGAM possesses the Statistical Association Property (SAP), as formalised by Wilming et al. (2023). This property ensures that features with zero statistical dependency on the label (even nonlinearly) receive zero attribution. In contrast, methods like LIME, SHAP, LRP, or Integrated Gradients can attribute importance to suppressor variables. This makes PatternGAM particularly well-suited for detecting issues like confounding or the "Clever Hans" effect (Lapuschkin et al., 2019) where a model exploits spurious correlations in the data.

Like most attribution methods, PatternGAM is not causal. In high-stakes domains like recidivism or healthcare, users could potentially overinterpret covariance-aware corrections as causal insight.

This is also true. However, again due to the statistical association property, PatternGAM is better suited as a screening tool to study causal associations than any existing feature attribution method known to us (e.g., LIME, SHAP, LRP, IG). While the dependencies uncovered by PatternGAM do not dissociate causal, anticausal, or confounded relationships, they provide a principled starting point for downstream causal analysis or auditing. We emphasise this distinction in the revised paper to help guide appropriate interpretation in high-stakes domains.

Questions:

1. The current formulation and evaluation of PatternGAM focus entirely on global explanations—i.e., producing a single explanation per class or model, rather than per input. Have you considered extending PatternGAM to the local explanation setting (e.g., similar in spirit to LIME or SHAP)? Specifically, could a covariance-aware linear projection be performed around a local neighborhood of an instance to produce a corrected explanation?

Thank you for this very thoughtful suggestion. In fact, we are exploring exactly that idea in a separate publication, for which we already published a preprint. We believe that the complexities that arise in the local setting due to the various design choices with respect to defining local neighbourhoods and covariances and the different possible applications and experiments (as the resulting methods are not restricted to tabular data but applicable to image classification tasks, for example), warrant a separate manuscript. Nevertheless, we will be happy to disclose our preprint here after the review process is completed for the reviewer’s interest (provided this is possible).

2. PatternGAM uses a post-hoc linear regression over shape function outputs to recover corrected attributions. While this is conceptually elegant and efficient, it may be sensitive to model complexity. How do you assess or control the fidelity of the linear approximation to the underlying model? Could this lead to under- or over-attribution in highly non-linear regions?

PatternGAM assumes, consistent with the GAM and NAM architectures, that the model prediction is a linear combination of nonlinearly transformed (per-feature) shape function outputs. This structural constraint ensures that a post-hoc linear regression over these outputs is not an approximation in the usual sense, but rather reflects the actual composition of the model. For more complex, non-additive interactions that cannot be expressed through individual or pairwise shape functions, the additive assumption may become limiting. In such cases, model performance may be degraded. We advocate that explanations should only be applied to well-performing models, where the output representation is then linearly separable.

3. Your work extends naturally to models with pairwise interactions (e.g., EBM, NAM with pairwise terms). However, it is unclear whether the method generalizes beyond this. Could the covariance correction be generalized to additive models with three-way or higher-order interaction terms? If so, how would the linear regression stage be modified to accommodate this structure?

Extending PatternGAM to handle higher-order interactions (e.g., three-way or more) is conceptually straightforward. These higher-order terms can be added as explicit components in the GAM or NAM architecture - either as handcrafted terms or learned sub-networks - and treated as additional features in both the model and the post-hoc PatternGAM attribution step. Importantly, PatternGAM does not require explicit construction of a high-dimensional covariance matrix. The attribution reduces to computing the covariance between each shape function output (whether a main effect or interaction term) and the target.

4. PatternGAM does not merely provide attributions—it offers corrected estimates that may be more robust to confounding. This opens up downstream opportunities. Could PatternGAM be used to guide feature selection (e.g., pruning suppressors), or to identify potential fairness issues in model behavior? Have you explored this in practice?

We agree that PatternGAM provides corrected explanations by disambiguating suppressors from informative features, and we appreciate the suggestion to explore its use for downstream tasks like feature selection or fairness auditing. However, we would like to clarify that by “correction” we refer strictly to correcting explanations, not altering the model itself. This distinction is crucial: Bayes-optimal models may rely on suppressor variables to improve predictive performance (as discussed by Wilming et al., 2023), and such variables should not necessarily be removed from the model.

5. FNI-EMD is a compelling metric that addresses a real weakness in common evaluation metrics (e.g., IMA, EMD) by ignoring false negatives. However, its formulation and examples are tightly coupled to spatial data (e.g., 2D tetromino patterns). Could FNI-EMD be extended to structured tabular data, categorical features, or time series? Or is it specific to grid-based visual domains?

While our empirical focus was on 2D spatial domains (e.g., XAI-TRIS), the FNI-EMD metric is not inherently limited to grid-based visual data. The core requirement for FNI-EMD is the ability to define a ground cost (distance) between features, which allows it to be extended to other domains, such as temporal distance for time series, Mahalanobis distance in feature space for structured tabular data, or Hamming distance for categorical features. We will clarify this distinction in the discussion section of the revised text, and will also mention the alternative ground metrics stated above for their respective problem domains in Section 3.2.

2. Interpretability trade-offs: No analysis is provided on how users interpret PatternGAM explanations compared to other baselines.

Thank you for the comment. We agree that understanding how users interpret explanations is important. While a formal user study is out of scope for this paper, we view it as a valuable direction for follow-up work and are actively exploring it. The standard NAM explanations (via shape functions) serve as a natural baseline, and our results demonstrate how PatternGAM corrects key misattributions caused by suppressor variables. We acknowledge that additional baselines (e.g., gradient-based or heuristic methods) may further contextualise our results, so we will include representative comparisons to such baselines in the final manuscript.

Paper Formatting Concerns:

Thank you for the suggestions. We will adopt these idea in the revised manuscript. For example, Figure 2 will outline the ground truth tetromino subplots, which are the same respective positions as in Figure 1. We will expand Section 3.1 to improve clarity by providing brief intuitive explanations alongside the formal derivations - particularly in the PatternGAM paragraph.

References:

Clark, Benedict, Rick Wilming, and Stefan Haufe. "XAI-TRIS: non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance." Machine Learning 113.9 (2024): 6871-6910.

Haufe, Stefan, et al. "Position: XAI needs formal notions of explanation correctness." Interpretable AI: Past, Present and Future, 2024.

Wilming, Rick, et al. "Theoretical behavior of XAI methods in the presence of suppressor variables." International Conference on Machine Learning. PMLR, 2023.

Thank you for the accurate and thoughtful summary. Please note that we deliberately avoid framing our method in terms of faithfulness, as this concept (as commonly used in XAI) can systematically assign importance to suppressor features — a concern raised in recent work (e.g., Wilming et al., 2023). This is also discussed by Haufe et al. (2024). Instead, we focus on producing explanations that better reflect statistical signal attribution, disentangling predictive utility from spurious correlations introduced by suppressors.

Weakness

1. Computational Overhead: Computing full covariance-based activation patterns, especially with interaction terms, may not scale well to very high-dimensional or real-time applications.

Thank you for raising this. While computational overhead is a valid concern for explanation methods, PatternGAM is designed to be lightweight in practice. PatternGAM reduces to computing the covariance between each scalar-valued feature (or interaction) output and the target, rather than requiring a full covariance matrix or matrix-vector product. This can be computed efficiently using standard statistics over the dataset. Moreover, unlike methods such as LIME or SHAP, PatternGAM does not involve training local surrogate models or computing Shapley value approximations. The method is purely post-hoc and closed-form.

If needed, approximate strategies such as diagonal/shrinkage covariance estimation, mini-batch statistics, or limiting the number of interaction terms (as we do via the FAST interaction selection algorithm) offer further scalability. We will make this more explicit by adding the above points to the discussion section in the revised version and provide bounds on the underlying computational complexity.

2. PatternGAM tends to produce "subset explanations"—highlighting only some relevant features. While this is justified, it could limit applicability in settings requiring complete ground-truth attribution.

Thank you for raising this point. We agree that PatternGAM can produce sparse or subset-style explanations, where only a subset of the relevant features receives non-zero attribution. This may reflect property of additive models like NAMs, where the model may shut down some redundant informative features through the function $f_i$, i.e. a feature’s shape function $f_i(x_i)$ collapses to a constant. Then, an informative feature $x_i$ would become an uninformative transformed feature $z_i$. Here, PatternGAM would correctly assign it zero attribution, even if that feature was relevant in the original data distribution. In this sense, sparsity in the explanation may not indicate incompleteness, but rather that the model has learned to ignore some redundant or weakly informative variables.

Questions:

Please check the weakness outlined above.

1. Does PatternGAM generalize to models beyond additive structures (e.g., deep networks, GNNs)?

Thank you for the question. PatternGAM, in its current form, is designed specifically for additive models. We see this as a first step toward bringing the soundness of the activation pattern framework to non-linear architectures. We chose (neural) additive models as a target because they are actively promoted as “intrinsically interpretable” models - yet, as we show, they are still vulnerable to suppressor-induced misinterpretations. PatternGAM addresses this by correcting how importance is attributed post-hoc, without requiring inverse mapping assumptions. While each subnet in a NAM can be made arbitrarily deep, and thus has some architectural flexibility, generalising PatternGAM to deep models or graph neural networks would require a principled way to extract structured, linearly separable representations suitable for pattern-based transformations - a challenge we see as promising future work.

References:

Wilming, Rick, et al. "Theoretical behavior of XAI methods in the presence of suppressor variables." International Conference on Machine Learning. PMLR, 2023.

Haufe, Stefan, et al. "Position: XAI needs formal notions of explanation correctness." Interpretable AI: Past, Present and Future, 2024.