Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons

We appreciate the reviewer’s insightful feedback. Please find our response below.

Improving the connection between the theoretical and empirical aspects of the paper

We agree that the connection between the theoretical and empirical aspects of this paper could be better articulated. Specifically, our findings demonstrate that generating a diverse set of configurations for minimal sufficient reasons is fundamentally intractable. This underscores the potential impracticality of computing such explanations, particularly for large neural networks with expansive input spaces in a post-hoc fashion. Furthermore, our intractability results remain valid even under significantly relaxed conditions, such as approximating the cardinality of the explanations or evaluating sufficiency using only a baseline. These theoretical insights correspond to the observed limitations of many post-hoc methods, which tend to be inefficient and, when applied to larger inputs, often generate subsets that are excessively large or lack faithfulness. This underscores the importance of integrating the learning of sufficient subsets directly during the training phase, thereby eliminating reliance on post-hoc computations and enabling the generation of subsets that are both efficient to generate, concise and faithful. We appreciate your suggestion and will ensure this point is more clearly emphasized in the revised text.

How do different maskings generalize to different forms of sufficiency?

We agree that this is an interesting point to explore. In response to this comment and another from reviewer NPb3, we will include a detailed discussion of this matter in the final draft, along with an additional experiment. We also emphasize that SST can be easily adapted to accommodate several forms of sufficiency by employing diverse types of masking simultaneously during training (a different one at each batch). However, we note that some sufficiency conditions already allow one form to naturally generalize to another, and this is an interesting aspect to discuss. The user who uses SST can determine the choice of sufficiency, and its corresponding masking, and the level of generalizability depends on the relationships between these forms.

In Table 2, both robust masking and probabilistic sampling operate within a bounded $\epsilon$ domain, whereas the baseline configuration lies significantly OOD, making its task inherently more challenging. This challenge is evidenced by the larger subset size (23.69%) generated by SST for the baseline compared to the robust and probabilistic settings. As a result, the baseline's larger sufficient reason generalizes well to the other configurations, achieving high faithfulness (98.91% for probabilistic and 98.38% for robust). In contrast, subsets generated by probabilistic and robust masking generalize effectively to each other (98.85% and 99.32%, respectively) but poorly to the baseline (11.82% and 8.16%, respectively), likely due to the baseline's significant OOD nature. This highlights the important role of $\epsilon$ perturbation choice, sampling distribution, and baseline characteristics in influencing the generalization between different maskings.

We appreciate you highlighting this interesting point, and we will discuss it more thoroughly in the final version.

Additional minor comments

Yes, you are correct regarding the cardinality mask size - this is indeed a typo in the plot. It should be updated from 0.5 to 50. Thank you for catching that!

We appreciate the reviewer’s detailed and insightful feedback. Please find our response below.

Practical implications of the work on interpretability

We thank the reviewer for this comment. While we do believe that our work provides significant contributions in the field, with many practical implications, we agree that our evaluations and analysis focus more on the systematic side of interpretability, emphasizing complexity, faithfulness, conciseness, and efficiency in explanations. However, other practical and human-centered aspects, such as human evaluations, the impact of simplified input settings, relation to bias detection (as noted by the reviewer), remain open for exploration. We will highlight these future directions in the final version.

A further discussion of the validity of the faithfulness metrics should be included

Thank you for this insightful point. We will include a more detailed discussion in the paper and address it in the limitations section as suggested. As with many explanation definitions, the concept of a sufficient reason can be elusive, with the “missingness” of the complement defined in various ways. Our work aims to address this by exploring multiple sufficiency forms. However, we acknowledge that no single definition “perfectly” captures a model’s internal workings, which also applies to faithfulness metrics assessing the sufficiency of subsets. This challenge is common in many XAI methods, such as SHAP [1], other additive attributions [2], and metrics like infidelity [3], where the treatment of “missing” features can vary, leading to differences in explanation definitions and metrics. We believe that the diverse sufficiency criteria explored in our work contribute significantly to a deeper understanding of this type of explanation. In the final version, we will expand the discussion to address the potential limitations of this diversity, which also extend to the faithfulness metrics.

Additional discussion on hyperparameters

We conducted an ablation study on the cardinality loss coefficient, as it was crucial in demonstrating the inherent trade-off between cardinality and faithfulness in SST, a central element of the framework. To address the reviewers' question about $\tau$ and $\alpha$, adjusting $\tau$ should not fundamentally affect convergence, as the model can adapt to different thresholds. However, extreme $\tau$ values can lead to convergence issues by pushing the model toward subsets that are too large or too small. Hence, we used the default $\tau = 0.5$. As for $\alpha$, as in adversarial training, smaller values discover adversarial examples closer to the input but increase training costs, while larger values find farther examples with lower costs.

We agree with the reviewer that more ablations on hyperparameters would be beneficial. In the final version, we will include additional experiments of varying hyperparameters, along with a sensitivity analysis. Thank you for highlighting this point.

Can SST be applied to a lower dimensional subspace?

Yes, the reviewer is correct that our approach can be applied to any simplified feature space, such as lower-dimensional subspaces or segmented input representations. This is an exciting direction with great potential, particularly in enhancing the human preference of explanations. In response to this and Reviewer NPb3's comment, we will include an experiment in the final paper demonstrating our method on a reduced, segmented input space and comparing it to the non-segmented setting. We will also emphasize exploring simplified input spaces as a key future research avenue.

How does the method's approach compare when applied to more interpretable models (e.g., decision trees)?

That’s an interesting question. Although our focus is on neural networks, obtaining cardinally minimal sufficient reasons is theoretically intractable for other models, including even "interpretable" ones like decision trees [4]. This result is surprising, as it reveals computational hardness in deriving explanations even for simple, interpretable models, though the complexity there is “only” NP-complete, which is less than neural networks ($\Sigma^P_2$-complete, etc.).

Since obtaining post-hoc explanations for decision trees is computationally challenging, training self-explaining models is an interesting research direction. However, as decision trees and similar interpretable models are less expressive than neural networks, identifying sufficient reasons for their predictions may be harder for the model to learn. This remains an open question. We agree that extending our work to other model types is a valuable research direction and will highlight this for future work.

We appreciate the reviewer’s valuable comments. Please find our response below.

Results on ImageNet and BERT

Although we recognize that the performance drop was indeed slightly more noticeable on ImageNet (as highlighted in the limitations), BERT-based models experienced less than a 1% decrease in accuracy across all benchmarks, while delivering significantly more faithful and notably more efficient explanations.

Why sufficiency explanations are useful, and the use of feature-level explanations

We appreciate the reviewer’s suggestion to enhance the background on sufficient explanations and we will address this in the final version. Sufficient explanations provide a distinct explainability framework compared to the more popular additive attribution methods like SHAP, LIME, or Integrated Gradients, offering insights often overlooked by these approaches, particularly around feature interactions and non-linear behaviors. For example, the top $k$ weighted coefficients in additive attributions do not indicate whether those features alone determine the prediction, a gap that sufficient explanations fill. The authors of Anchors [1] demonstrate that such explanations often offer more intuitive and human-preferred insights than traditional additive ones.

We focus on direct sufficient subsets of the input space, aligning with methods tackling the same task [2-5], while offering advantages like: (1) detailed, localized insights, (2) reduced arbitrary segmentation issues, (3) minimized information loss, and (4) improved faithfulness of predictions. While our results capture the minimal input subset for predictions, we agree with the reviewer that extending the framework to higher-dimensional spaces could enhance certain aspects of interpretability, particularly from a human perspective. This opens opportunities for future work. Importantly, our method can be applied to any feature space.

In response to this and Reviewer GpEy's comment, we will add an experiment to the final paper applying our method to a reduced, segmented input space and highlight the value of exploring additional simplified input spaces as a valuable direction for future work.

What’s stopping the model from learning a “cheating” solution that always produces the same subset?

We agree that this is an important point. Like many deep learning tasks, our framework risks the model exploiting "shortcuts" or converging to undesirable local minima. However, our optimization objective explicitly avoids favoring the mentioned configuration, making such convergence very unlikely.

First, it is important to emphasize that the explanations generated by our approach are inherently local rather than global. For each input, the model identifies a unique sufficient subset specifically tailored to that input. Because different inputs usually require substantially different subsets as minimal sufficient reasons for their predictions, a model that consistently produces the same subset would inherently fail to be faithful, contradicting our objective of optimizing for faithfulness.

Furthermore, as shown in the figures in the main text and appendix, subsets generated for different inputs vary significantly within the same benchmark, demonstrating this issue does not arise in practice. We ran an initial experiment on CIFAR10 with robust masking (average explanation size: 12.99%) to support this: 0.07% of pixels appeared in 84% of explanations, 0.14% in 70–80%, 28% in less than 10%, and the remaining 72% varied from 10–80%. While some overlap is observed (which is expected, as the important pixels typically appear near the center of the image), the explanations overall exhibit significant variation.