Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups

Thank you for the feedback! In the following response, we clarify the posed questions and commit to adjustments as needed.

Experimental Setup Clarifications

• What features are groups based on? Groups are simply subsets of the input features (the $x_{G_i}$ from Definition 1). A SANN takes such groups and embeds them in a latent space (the $h(x_{G_i})$ from Definition 1) with a neural network to make predictions. In our experiments, $x_{G_i}$ manifests as subsets of image patches and text tokens (line 261-262). We will make this more explicit by adding the following statement to Definition 1: “We refer to $x_{G_i}$, the subset of input features of $x$ corresponding to the subset $G_i$ as a group of features of $x$.”
• Datasets and Metrics. On line 256-262, we gave the general categorizations of the tasks (image classification for ImageNet, image regression for CosmoGrid, and text classification for MultiRC), with further details in Appendix C. To summarize: The classification tasks of ImageNet/MultiRC are to predict the right object in the image or answer the multiple choice question (Appendix C.1.1 and C.1.3). The regression task for CosmoGrid is to predict cosmological parameters of the universe from telescope images (Appendix C.1.2), where the MSE (Mean Square Error) measures how much the predicted cosmological parameters differ from true values (lower is closer and thus better).

SOP for Model Debugging (RQ7)

The goal of debugging is to identify a “bug”, or the reason behind errors. Here, we find that when SOP is correct, it tends to be using more of the object, whereas when SOP is incorrect, it tends to use more of the background. This analysis shows that one potential bug behind the errors of SOP is when the model looks at the background as an inaccurate proxy for the object. This finding is only reliable because of SOP’s (and other SANNs) faithfulness guarantee–since predictions depend solely on the selected feature groups. Without the faithfulness guarantee, one then cannot be confident if the prediction depends on the explanation. Since SOP has more semantic objects and backgrounds in its groups (from RQ5), the analysis is applicable to more SOP predictions than other SANNs.

Clarifying the Cosmology Study (RQ8)

The mapping between Section 4.3 and RQ8 is correct. Specifically, RQ8 asks a scientific question on the dependency of network predictions on cosmological structures. However, analyzing these cosmological structures with SOP only makes sense if the SOP groups contain such cosmological structures to begin with. This fact was established previously in Section 4.3 (RQ5), which measured the semantic coherency of SOP groups in various settings including cosmological structures in CosmoGrid, the setting of RQ8. That is why RQ8 references Section 4.3. We will update the reference to refer specifically to RQ5.

In the final conclusion, we called the result “meaningful” as the conclusion was understandable to experts, and “scientific discovery” as these are relations that experts want to understand but currently do not. We did not intend for this to be a final irrefutable claim: rather, it is an initial step into the unknown that allows follow-up work to independently confirm or investigate more deeply. We will adjust the wording to reflect this. These claims were made jointly with expert cosmologist collaborators, who helped us write background, justification, and conclusions of RQ8.

Relation to Object Factorization

At a high level, both SOP and methods such as slot attention use attention mechanisms to group input features without direct group supervision. However, the goals (and corresponding design decisions) are quite different.

SOP forms feature groups that are beneficial for the end-to-end prediction task, without trying to reconstruct the original input. In contrast, techniques like slot attention try to break an input into its parts by learning groups that can reconstruct the input. Slot attention forces competition between groups to contain disjoint components and get coverage over the entire input, whereas SOP allows for groups to both overlap or not cover irrelevant features. Overlapping information (e.g. in Figure 1) can benefit prediction, i.e. when objects have multiple relevant contexts/groups in the image. We will add this discussion to the related work.

Other Comments

• We will correct the grammatical error in Line 201, and enlarge Figure 3 (a larger version can be found in Figure 4 in https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf ).
• For line 326, we meant for this to refer to human understandability being improved if the explanations are semantic, and will adjust accordingly.
• Lastly, we can certainly swap RQs to be fully in the appendix and enrich the results for other RQs in the main paper. If you can let us know which RQs should be moved, we can adjust accordingly.

Thank you for your feedback. We respond to your questions as follows:

Backing up the Claim that SOP’s Groups are Superior

We would like to clarify a misunderstanding of the experiments: In referring to semantic utility, we believe the reviewer is confusing the claims on group quality with the wrong set of experiments.

Specifically, experiments showing that SOP’s groups are semantically superior are in Section 4.3, which validate the semantic coherence of the groups. These experiments measure the semantic meaning of the groups, and evaluate all baselines on all datasets, including cosmology.

We believe the reviewer is misinterpreting Section 4.4 as an evaluation of the groups. Instead, Section 4.4 highlights how these semantic groups can be utilized in downstream tasks, not to measure groups. In practice, practitioners will want to use and interpret the best performing model and have an easier time interpreting results that are more semantically aligned, which is SOP as validated respectively in Sections 4.2 and 4.3. For example, our cosmologist collaborators are only interested in interpreting models that can predict the ground truth parameters with low error to obtain trustworthy insights for scientific discovery.

Clarification for Experiments Metrics

• Error Metrics. To our knowledge, our error metrics are within standard practice in the literature for measuring the performance in these tasks. ImageNet is almost universally evaluated with top-1 error/accuracy. Cosmogrid uses mean-squared error as it is a regression problem, and is also consistent with prior work. Reporting F1/AUC is not common in these settings. MultiRC can be evaluated with either accuracy or F1 score, and many before us have used accuracy without a significant difference. We reported accuracy to be consistent with ImageNet, but can easily change this to F1 (the results and findings are the same).

• Exact formulation of IoU-based Semantic Utility. At the end of RQ5, we direct the reader to Appendix C.4.1 (line 1550) for detailed descriptions including the exact formulation for IOU, how it is calculated, and the annotations used, which are object segmentations and human-annotated explanations. Semantic groups that align with objects in images or structures in mass maps are easier for humans to interpret in downstream analysis, as in RQ7/RQ8.

Significance of Theoretical Claims

While the idea that grouping is more expressive than per-feature methods may seem intuitive, this has remained a heuristic with no formal proof. Our work provides the first formal mathematical proof of this key property. Furthermore, our theorem provides two key insights that go beyond this intuition. First, our theorem proves that SANNs require only a finite number of groups, contrasting other theoretical DL results that have classically assumed infinite width layers to achieve good performance. Second, our theory proves that it is impossible for per-feature SANNs to achieve good performance. Until now, it was not known whether per-feature SANNs could be improved with more data, larger models, or better training.

Why do some baselines have high error?

In our experimental setup, we carefully controlled for all baselines to have equal compute costs and equal sparsity when measuring accuracy. This is critical in order to attribute better performance to the SANN approach rather than more computation or more features. In Appendix C.3 we describe the common setup: all methods use at most 20 forward passes with 80% sparsity (keeping 20% features). The high error rates arise from baselines failing to be efficient with respect to compute or sparsity. RISE-F typically requires thousands of random masks to be effective, which is impractical for ImageNet inference. The performance of XDNN and BCos-F is due to their design depending on having all the features (0% sparsity) to make accurate predictions.

How is the number of groups controlled?

The maximum number of groups is a hyperparameter that can be selected based on the user’s available computational resources. We analyzed how different maximum numbers of groups affect performance on ImageNet in Figure 1 in https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf . We see that performance increases with more groups until it plateaus at 5 groups, indicating that our model can potentially be 4x more efficient!

Request for pseudo-code and code repository

The requested pseudo-code and code repository are already in the submission. The pseudo-code is in Algorithm 1 at the top of Appendix B (page 24, line 1265-1279). The code repository is included in the supplementary material.

Other comments:

• We will add the backbone performance to Table 1, which is 0.097 error for ImageNet, 0.00869 MSE for Cosmogrid, and 0.318 error for MultiRC.
• We will add a table of contents to add structure and clarity to the Appendix.

Thank you for your feedback. We provide an additional PDF at https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf for additional figures and will respond to your questions below.

Comparison with Additional Shapley-Based Baselines

We first would like to clarify a key distinction between SANNs and the referenced line of Shapley-based methods. Specifically, these Shapley-based methods (SBMs) are not Self-Attributing Neural Networks (SANNs). Instead, these methods are faster approximations of the SHAP baseline that we already compare to. The goal of these SBMs is to efficiently approximate post-hoc explanation Shapley values, with later works (FastSHAP, ViT-Shapley, Auto-Gnothi) training additional surrogate models to estimate Shapley values. AutoGnothi, for example, trains multiple side networks to generate approximations of Shapley explanations. These SBM explanations are crucially never passed into the backbone to make a prediction. In contrast, in a SANN, the features in an explanation are directly passed into the backbone to guarantee that predictions rely solely on the explanation. Therefore, the explanations of AutoGnothi and other Shapley-based methods are more similar to post-hoc explanations and do not fall into the category of SANNs. This fact is reflected in SBM experiments: all the papers of FastShap, ViT-Shapley, and Auto-Gnothi only compare to post-hoc explanation methods and do not compare to any SANNs, as a direct comparison is not possible.

In our paper, we provide comparisons to SANN variants of post-hoc methods. Since we already compared to a SANN variant of SHAP (in Table 1, denoted SHAP-F), for completeness, we ran analogous comparisons with SBMs using their released codebases. Since the suggested methods involve training additional side networks, for fair comparison, we ensured that all baselines had equal or greater training resources than SOP. We show the results in Table 1 in the additional PDF, where we find that their performance as a SANN is actually even worse than SHAP, and consequently worse than SOP. Intuitively, this result is not too surprising since these methods are faster but more inaccurate approximations of SHAP, trading off accuracy of the Shapley value for speed. KernelSHAP was too expensive to be fairly run as a baseline, as it requires a minimum number of forward passes at least equal to the number of features to find the solution to their loss function in Theorem 2 in their paper and code [2], making it an order of magnitude more expensive than all other baselines.

On Computational Cost of SOP

Since the reviewer brings up computational cost as a potential weakness, we point out that our SANN experiments are orders of magnitude larger scale than those of the referenced SBMs. The AutoGnothi and ViT-Shapley papers conduct experiments on small scale datasets such as ImageNette, which only has 10 easily distinguishable classes where it is easy to get 100% accuracy. In contrast, we train SANNs on full ImageNet (1000 classes!), where SOP achieves SOTA performance with only one epoch of training. Since we are evaluating SBMs in a significantly larger setting than originally proposed, we checked the accuracy of surrogate models to ensure these baselines were trained properly. We confirmed that the resulting surrogate models trained on masked inputs have comparable accuracy on ImageNet to other work in the literature [1, Figure 10] that also trains models on similarly masked inputs, so we are certain these baselines were sufficiently trained.

If the reviewer is still concerned about cost, we have also included an ablation study (Figure 1 in the additional PDF) showing that SOP performance on ImageNet saturates at just 5 groups, meaning we can further reduce inference costs by an additional factor of 4 beyond what we originally reported.

[1] Jain et al. Missingness Bias in Model Debugging. ICLR 2022.

Human Distinction Task With More Samples

First, we point out that our human distinction task follows the exact same protocol as the original HIVE paper published at ECCV [2a], which uses 10 examples for evaluating human distinction tasks [2b]. We believe this to be partly because evaluating the task on a single example requires significant human effort, with the original study consisting of hundreds of evaluator tasks. We have increased the study to 50 examples, which amounts to thousands of evaluator tasks. In summary, we found results consistent with what we originally reported previously (that SOP is among the best, but that the distinction test cannot distinguish between the best), but with slightly smaller error bars and smaller differences between methods. The expanded study is shown in Figure 3 in the additional PDF.

[2a] Kim et al. HIVE: Evaluating the Human Interpretability of Visual Explanations. ECCV 2022.

[2b] https://github.com/princetonvisualai/HIVE/blob/main/materials/HIVE_suppmat.pdf

Thank you so much for the valuable and encouraging review! We provide an additional PDF at https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf for additional figures.

Explaining the Deletion Performance of SOP.

To explain how SOPs usage of multiple groups can affect the deletion, we show the average deletion curves for SOP in Figure 2 in the linked PDF. We see that even as we delete features from groups, SOP is able to maintain relatively high performance in some cases, resulting in a worse deletion score. Such behavior is natural because the training objective in SOP encourages the group selector to select highly predictive groups, and multiple groups can compensate for the information missing in another group.

What is the computational complexity of SOP?

The cost of SOP (running time and memory) is dominated by the forward passes through the backbone predictor $h$, which can be seen in the definition of $f$ in Section 3. Therefore, when using $m$ groups, the cost of an SOP forward pass is equivalent to $m$ forward passes through the backbone. The costs of the group generator and the group selector are negligible in comparison, since each one is a much smaller attention module on sequences of $m$ inputs. We will make this cost explicit at the end of Section 3.

In our experiments, we control computation fairly across all baselines, which is discussed at length in Appendix C.2 & C.3. All methods can use at most 20 forward passes per inference with the exception of Archipelago which requires quadratic forward passes (Appendix C.3 line 1531-1533). For SOP, this amounts to m=20 groups and thus 20 forward passes per inference (Appendix C.2.2 line 1462-1463). SOP can match or beat all other baselines with equal compute.

A detailed summary of computation is in Table 2 in the linked PDF, which we will add to Table 2 of the main paper to make this information more visible.

Lastly, the training needed to fine-tune SOP modules (Appendix C.2.2 line 1464-1465) are minimal compared to training the backbone (e.g. one epoch for ImageNet).

Questions about Group Diversity.

Multihead attention and explicit diversity regularization could potentially change group diversity. We found that simply using more attention heads in the group generator actually helped create more diverse groups, an observation that we briefly mentioned in Appendix C.2.2 line 1462-1462. Concretely, we observed that using four heads in the group generator results in 32% fewer overlapping patches in groups than one head on ImageNet.

Comparing SOP to Baselines with the Same Backbones.

We confirm that SOP is already compared to baselines with the same backbones whenever possible. Specifically, we compared 10 model-agnostic baselines that use the exact same backbone model as SOP, and 4 baselines that are architecture specific. This is summarized in the main paper in Table 1 (see “Model-Agnostic” column) and discussed at length in Appendix C.3. We included architecture-specific baselines as readers wanted to see such comparisons.

Generalizability of Polynomials in the Theory.

We note that our theory tackles polynomials in the most general setting, with no restrictions on the type or degree of polynomials, and therefore has broad implications. Thanks to the Stone-Weierstrass theorem, every continuous function can be uniformly approximated by a polynomial function. Since our result applies to any polynomial, it also applies to those that uniformly approximate real-world data patterns.

Sensitivity to Sparsity / Number of Groups.

• Sparsity Analysis. We have already done this sparsity analysis in Section 4.2 RQ2 (Figure 3), with further analysis in Appendix C.4.2 (Figure 18).

• Number of Groups Analysis. In Figure 1 in the linked PDF, we measure how the number of groups affects accuracy for ImageNet. Naturally, having more groups improves accuracy, and it saturates at 5 groups, suggesting that SOP can actually be 4x more efficient than originally reported.

Further Visualization & Analysis of Groups.

We have already included visualizations in Figures 5+7 in the main paper as well as Figure 11~15 in the appendix for more visual examples, and RQ5 and RQ6 do a semantic analysis of said groups. Furthermore, RQ7 and RQ8 provide insights on how different features groups learned with SOP affect predictions.

More Context for Cosmology Study.

We referenced additional detailed and accessible cosmology background in Appendix D, where we introduce the problem and discuss the kinds of findings and groups cosmologists find interesting.

Biases in Pretrained Models.

Pretrained models are known to encode a range of social biases including racism and sexism. One could utilize algorithms that edit models for fairness or debiasing models. Such biases equally affect all baselines in our work using the same pretrained model.