Intrinsic User-Centric Interpretability through Global Mixture of Experts

We kindly thank Reviewer 4 for this careful evaluation of our work and greatly appreciate the review, highlighting our paper’s strengths in presentation, “novel” methods, and “clear and solid” motivation. We strongly agree with all points raised.

We have addressed each in a point-by-point response below and feel that the resulting edits have further improved the manuscript.

Potential over-claiming due to unclear definitions of certain axes of approach comparisons. We thank the reviewer for the suggestion that will greatly improve the clarity of our paper’s contribution. We have now included a full taxonomy of terms based on definitions of explanation criteria derived from related literature, which have been included in Appendix A. We have also clarified the claims in the introduction and tied in more literature in the “Interpretability Foundations” section.

The reviewer also asks for justification of our categorization of “faithfulness” and “coverage” in light of the InterpretCC approaches. The definition of explanation faithfulness is the “alignment between the explanation and the model’s inherent decision making process (ground truth)” (Lyu et al. 2022, Dasgupta et al. 2022). As the explanation in InterpretCC FG (selected features from the discriminator layer) and InterpretCC GR (selected concepts from the discriminator layer) are directly used in the prediction of the instance, we believe this makes the explanation faithful to the model’s decision making process. The definition of explanation coverage is the “the extent to which an explanation accounts for the features or concepts utilized by a model to make a prediction” (Ali et al. 2023). Both InterpretCC FG and GR provide explanations that explicitly select a subset of the feature space, and only use the features that are selected in the prediction. Therefore, both models have full coverage (the explanation covers all the features used in the model’s decision process).

Comparison to other intrinsic methods that use expert knowledge. After a thorough literature review, we do not find a similar intrinsic approach that uses expert knowledge to separate the fully separate feature space into concepts. There are a few models, primarily in the vision domain, where concepts are defined through expert-selected prototype examples or rules (Koh et al. 2020, Konstantinov et al. 2024). We add these to the discussion in Sec. 2. A stronger comparison point is to expert-specified in-hoc approaches (i.e. concept activation vectors) that have been discussed already in the background section.

Regarding R4’s concern of fair comparison, all explainability approaches we compare ICC to (i.e. SENN, NAM, LIME, SHAP, IG, TCAV, and the newly added FRESH) use the exact same input features and modeling setting as ICC with real world use cases, creating a fair comparison between both intrinsic and post-hoc approaches. If anything, methods like SENN Concepts, D-TCAV, or Sum-of-Parts which optimize the selection of feature groupings would intuitively have a performance advantage over expert-specified knowledge, as ICC’s grouping is centered on user actionability and usefulness instead of optimizing directly for predictive potential.

Settings where domain knowledge cannot be easily found. Indeed, our method requires specification domain knowledge, i.e. expert input. In settings like the breast cancer experiments, clinical knowledge that is prediction task-specific is required to interpret the results of the tests into even more meaningful medical categories, although we were able to group by cell and by lab test. However, LLMs continue to get better at grouping features into concepts, as evidenced by our experiment in Table 4 with a GPT 4 model, where without expert feedback, concepts that were selected were both human-understandable and effective at prediction in comparison to expert-extracted categories. We believe the abilities of LLMs to group descriptions of input features or raw input streams into meaningful concepts will only increase the effectiveness of the ICC models. We have added a nuanced discussion of this into work in Sec. 6.

Additionally, for modalities like raw time series, it is very hard to have human interpretable knowledge. Graph based models could be very useful in the InterpretCC architecture for raw time series data, with a natural extension in the discriminator stage. The message passing graph network and concept activation vector approach showcased with RIPPLE (Asadi et al. 2023) could be used to define a sparse adjacency matrix as opposed to a sparse vector on the input. This could then relate to interaction-based subnetworks, referring to multiple features in each concept used in the explanation.

[continued in the next comment]

We kindly thank Reviewer 3 for the careful evaluation of our work and greatly appreciate the strongly positive review, highlighting our presentation, soundness and contribution, as well as the “great plus” of the user evaluation. We agree with all points raised.

We have addressed each in a point-by-point response below and feel that the resulting edits have further improved the manuscript.

Interaction of features is important for explanations in real world settings. We agree, we discuss this more now in Appendix I.5. With the IntepretCC Group Routing model, the design of concepts inherently utilize the shared interactions of related features in the predictive subnetworks. Similarly, for the Feature Gating case, selecting the features first and then having a predictive model see all the “important” features together is an advantage that takes into account the features’ shared predictive potential. This is in contrast to other intrinsic approaches like NAM or SENN, which assign linear scores to each feature individually. While InterpretCC FG selects the important features automatically (and predicts based on their shared interactions), InterpretCC GR uses user-specified or LLM-specified concepts, creating explanations with shared feature interactions that are by-design actionable for the user. We believe in expert-specified concepts because if the explanation is not actionable, it might not be useful for anything other than model auditing.

Rephrasing of the “because and only because” terminology used to describe InterpretCC explanations, due to association and not causal claims. It is possible (rare case) that the discriminator selects regularity and video watching but does not fully use them in the prediction. The model is trained to explicitly avoid this behavior (harsh sparsity constraints, minimizing predictive losses), but it is still possible. As per the reviewer’s suggestion, we have revised the sentence on line 87 to be: "The student’s regularity and video watching behavior were the only two aspects selected as important for the student's prediction of passing the course, and the model did not use any other aspects to make this prediction."

Additional details on how to group features in challenging cases (i.e. when a certain feature could belong in two groups). Great suggestion, we have incorporated a discussion of this in Appendix C. In InterpretCC, we design towards the user's actionability of the resulting explanation. Therefore, if the knowledge that a feature is important can lead to a specific action, and if this action is the same one that should be taken for other features, then those features should be grouped together. From the modeling perspective, grouping features together in a concept means that their shared predictive potential should be leveraged, and likely this is more important for one type of features than another. We maintain that a feature should not be placed in multiple groups to have the faithfulness guarantees that are a strength of InterpretCC. However, in a rare and specific case where two actions must be taken based on the feature, or it is too difficult to decide which concept the feature belongs in, it is always possible to 1) put that feature in its own subnetwork (and have it be selected alongside any of the other feature groups), or 2) combine the two subnetworks into a larger concept.

Possibility to use graph models to extend the explainability from single feature based to feature interaction based. Yes, definitely. Graph based models can be very useful in the InterpretCC architecture, as a natural extension in the discriminator stage. For example, the message passing graph network and concept activation vector approach showcased in RIPPLE (Asadi et al. 2023) could be used to define a sparse adjacency matrix as opposed to a sparse vector on the input. This could then relate to interaction-based subnetworks, referring to multiple features in each concept used in the explanation. Alternatively, graph models like Zhang et al. 2022 could be used in the predictive module stage, where each subnetwork is a graph model only focused on the specific modalities or features passed into the subnetwork. Using the sparseness enabled by the discriminator layers, only a few of the graph models would be activated for each point’s prediction. We have added this discussion of InterpretCC’s extension to graph models into Appendix I.5.

[continued in next comment]

D) We envision InterpretCC to work increasingly on LLM extracted features instead of requiring human expert effort. LLMs/LMMs are getting very good at extracting features from raw data (Malberg et al. 2024, Baddour et al. 2024), and as the field moves forward, they will only continue to get better. IntepretCC will be further strengthened by these upcoming advances.

Explanations produced with InterpretCC can be misleading/unfaithful. We have now included a discussion (Sec. 6) of the risks of InterpretCC learning correlations that are not intuitive (superficial) in an explanation to make a prediction. By prioritizing human-understandability, our approach could leverage explanations as a tool for auditing models (Fabri et al., 2021; Yadav et al., 2022), with the potential for InterpretCC explanations to reveal a superficial pattern that prompts human expert intervention instead of incorrect model use.

Limited analysis of impact of sparsity threshold in the main paper. We have expanded upon the Appendix experiments with additional results on the breast cancer dataset, and moved a discussion into the main paper of the relationship of GS threshold, Tau, and model performance. In Appendix Fig. 6, we had reported the average of several experiment runs (with several seeds). We chose not to show the confidence intervals as it crowds the image and makes it less readable. We are now analyzing significant CIs in Appendix D.2 as well. The revised Fig. 6 additionally includes a new analysis on the relationship between feature sparsity and threshold across both datasets, as well as 8 plots on the relationship between threshold, tau, and model performance. Fig. 13 and 15 granularly show activations across subnetworks for different domains. Although the 95% CIs overlap for all variations, meaning that the performance is not significantly impacted by changing Tau and the GS threshold, the average performance of settings with a high Tau (around 10) and higher thresholds (between 0.7 to 0.8) maintain sparsity, performance, and stability.

Limitations of user study. The reviewer comments that the user study was done on only four test samples from a single dataset, potentially limiting the generalizability of the findings.

Of the 8 datasets, we chose the education domain and the DSP course specifically because it has been well studied in previous papers (Kidzinsk et al. 2016, Boroujeni et al. 2018, Swamy et al. 2022). The education domain is general enough to allow us to recruit numerous domain experts (in this case, teachers) with a large diversity of profiles and experience, as shown by the socio-economic data we collected, such as their teaching level (see Appendix Fig. 7). This increases the diversity of opinions, hence the generalizability of our findings. Moreover, this dataset offers a concrete and realistic use case where key criteria like actionability and usefulness are easy to assess. The tabular dataset about breast cancer would not allow such diversity in interpretations as it requires niche knowledge, while the synthetic dataset is too abstract for the participants to grasp the pros and cons of each explanation. Finally, the two textual datasets were not suitable for a study, since they are not adapted to the SENN and NAM methods. To our knowledge, this is one of the first user studies done solely on interpretable-by-design models and user preferences, so we wanted to select a setting enabling us to include all approaches.

Second, we chose not to include more samples due to the length of the study. Comparing 4 explanations across 5 criteria for 4 samples took nearly 30 minutes for the teachers participating in the study; an excessively long study would decrease the quality of their annotations after several samples. We had experimented with a larger quantity of randomly chosen samples (even up to 10 entries) in 8 pilot experiments, and found strong opinions that it was an extremely mentally taxing task to complete. Therefore, we chose to keep the same samples (chosen randomly) with a more diverse study population.

Regarding how we chose to visualize the explanations produced by each method: as we underline in the discussion section, we cautiously post-processed the explanations of SENN and NAM to provide them in a format understandable for a non-technical audience and iterated on the visualizations using a human-centered design process, through extensive iteration with 8 pilot participants. This iterative design resulted in visualization being optimized for teacher perceptions of each method. We note that any imbalance in wording is not necessarily in favor of our method; for instance, users found SENN's explanations more complete than InterpretCC (see Fig. 5).

We feel these suggestions have strongly improved the paper and we hope that our additions have increased confidence in the paper.

We kindly thank R2 for the careful evaluation of our work and are happy that the reviewer found the writing clarity clear and easy to follow, the experimental analysis thorough over many different datasets, and the explanations useful without sacrificing performance.

We have been able to address each concern in the point-by-point response below and feel that the resulting edits have further improved the manuscript.

Comparison to explain-then-predict and extractive rationale baselines: We now provide experiments with a popular extractive rationale, explain-then-predict baseline (FRESH, ACL 2020), compared to InterpretCC FG, InterpretCC GR for both text datasets (SST, AG News), discussed now in Table 2, Sec. 5.1 and Appendix I.4. We tuned the FRESH model over rationale length and learning rate, and ran experiments with 5 random seeds with the exact architecture used in the paper. Notably, FRESH experiments use BERT models, while ICC experiments use the smaller DistillBERT model. For SST, FRESH uses a 30% rationale size with an average accuracy of 82.05% +/- 0.56%, where ICC FG has 88.21% +/- 3.41%, ICC TopK has 92.98% +/- 0.88%, and ICC GR has 91.75% +/- 1.86%, which are significantly more performant than FRESH. As SST has 7 words on average, ICC FG selects a larger explanation size, on average 3 or 4 words (indicated in Figure 2), but maintains a higher accuracy. For AG News, FRESH uses a 20% average rational length, with performance of 88.73% +/- 0.69%. This is comparable to the ICC GR numbers 90.35% +/- 1.07% and the the ICC Top K routing approach with 87.25% +/- 2.48%, and more performant than the ICC FG approach 85.72% +/- 5.31% (although the high variation in the FG approach is to be noted), and the 95% CIs overlap showing that the approaches are similar.

In Appendix I.4, we discuss key differences between InterpretCC and extractive rationale methods like FRESH (Jain et al., 2020), summarized as follows: Task Generalization: FRESH requires training three separate models sequentially for each task, whereas InterpretCC trains a single model with two parallel parts. Bias in Saliency: FRESH relies on different methods to establish initial importance scores (e.g., LIME, attention, gradients), unlike InterpretCC, which learns importances directly. Model Complexity: FRESH's 3 models are larger, more complex, and harder to train compared to InterpretCC's simpler architecture. Explanation Format: Importantly, FRESH selects contiguous text spans as rationales, suitable for text but not for tabular or time-series data, where InterpretCC maps features to concepts, offering a flexible explanation format for multiple modalities. Due to this limitation, we did not extend the implementation of FRESH to other modality experiments (time-series, tabular) as using contiguous spans of features as explanations would be fundamentally unsuited to the data format.

A nuanced discussion on the requirement for interpretable features is needed: We have now added a statement regarding evaluation with interpretable features in Sec. 5, and a discussion of this in Appendix I.5 and Sec. 6. While several of the benchmarks we examined used expert-inspired features (i.e. education datasets), others like the healthcare datasets used unmodified lab data (where measurements still are human interpretable). The labeling of these features does make InterpretCC more useful; the education setting is showcased in the user study. However, we would like to address this concern with a few points:

A) For our experiments in the text domain, we did not use human-interpretable features, but worked with the unmodified text. Text, like images, videos, or speech data, is an inherently human interpretable domain (while clickstreams and lab measurements can often be more difficult to interpret). We also saw ICC perform strongly on the OpenXAI synthetic data, where there is no human interpretable meaning.

B) It is possible for InterpretCC to use raw time series data, simply specifying the “concepts” as fixed or relative time intervals (analogous to anomaly detection), enabling users to identify which parts of the time series were used in the prediction. Using raw clickstreams directly is rare in human-oriented domains; these domains typically have interpretable features (education, heath, etc.), or can extract interpretations directly (e.g. Asadi et al. AAAI 2023). We now discuss the case for raw time series in Appendix I.5.

C) As noted in line 99, we do not address the vision domain, as strong modality-specific interpretability methods already exist (Böhle et al., 2022; You et al., 2023; Donnelly et al., 2022; Thomas et al., 2023). While InterpretCC could be adapted for vision tasks, such as using object detection for grouping, these approaches would be less user-friendly than existing methods. Our architecture specifically targets gaps in other modalities.

[continued in next comment]

We kindly thank Reviewer 1 for the careful evaluation of our work and greatly appreciate the reviewer’s positive review of InterpretCC’s high originality, quality, and clarity with “a very thorough experimental section and results”. We strongly agree with all points raised. We have addressed each in a point-by-point response below and feel that the resulting edits have further improved the manuscript.

ICC feature gating is “almost the same as SENN feature except with a sparse mask”. We find this an important point to clarify, as there is a significant difference in the underlying design between the two models.

SENN has three components: a concept encoder that transforms the input into a small set of interpretable basis features, an input-dependent parametrizer that generates relevance scores, and an aggregation function that combines them. InterpretCC-FG is a much simplified version of this architecture, where one model is trained end to end with 2 modules instead of 3 models working together. We select important features (via the discriminator network with Gumbel Softmax) and only pass those to the predictive model.

Other than the sparsity and a different training architecture, a key difference is in how the resulting predictive score is obtained in inference. The parametrizer model in SENN Features directly generates scores that are linearly combined for all features. In InterpretCC, the importance selection happens first (reducing the need for extra computation) and the selected features are passed together to a neural network; therefore our predictive module leverages cross-feature interactions in a simpler and more intuitive way, instead of having a score for each feature. We are happy to clarify this in the paper in Sec. 5.

ICC group routing might be inefficient when the number of groups significantly increases, since the model complexity will increase as well. Indeed, model size increases with more groups (which require more subnetworks). However the complexity remains the same, as the same discriminator network routing mechanism can simply output a larger vector for a larger number of groups (linear scaling). The human understandability also remains the same (2 groups chosen out of 25 is the same as 2 groups chosen out of 6). Another factor to consider is that with experts specifying the groupings, a large number of groups is less likely to occur.

Further investigation of the effects of sparsity, specifically on Guble Softmax temperature (“ICC sparsity depends on the temperature of the Gumbel Softmax”). Our initial submission already includes exploration of this in Appendix Figure 6, exploring the effects of changing Tau and the Threshold parameters on the accuracy for the most popular education dataset. We have now also included additional hyperparameter variation experiments for the healthcare and text cases in Appendix D.2, as well as an extended discussion about this in the main paper.

The updated Appendix Fig. 6 includes a detailed analysis of the effects of modifying Tau and Threshold parameters on the performance of the FG and GR models across both education (time-series) and healthcare (tabular) datasets. We found that while the performance of InterpretCC has overlapping 95% CIs while changing parameters, certain parameter settings have higher variability than others. For both education and health tasks, a Tau of 10 and a Gumbel-Softmax threshold of around 0.7 to 0.8 are performant, sparse in activated features, and relatively stable.

Use of groupings of features as input for the discriminator network. The reviewer’s understanding is correct, the discriminator is not aware of the feature groups explicitly. However, since the discriminator is trained end-to-end along with the predictive subnetworks, it learns the groupings through the routing logic. We make this more clear in Sec 3.2.

[continued from “General Response 1/2”]

REVIEWER 3 provided a very positive review stating “the unique strength of this paper is to put user-centric design into intrinsic explanation models”, and praising the decision to “extend models from vision to other modalities”, the “systematic evaluation” with “well-defined metrics”, and the addition of the user study. R3 recommends acceptance with a score of 8 (“accept, good paper”). The reviewer requested the following:

1. A discussion on the interactions of features in the InterpretCC framework, and the possibility to use graph based models. InterpretCC requests experts (human or LLM) to group features with strong interactions into concepts so the predictive subnetwork can make a decision leveraging the features’ shared insights. Graph models can be very useful here, either adapting the discriminator network with a message passing network like RAINDROP (Zhang et al. 2023) that would output a sparse adjacency matrix as opposed to a sparse vector, or in using graph models as individual subnetworks.
2. A revision of the “if and only if” statement in the introduction. Great point – indeed, even if InterpretCC selects certain features and these are passed to the prediction module, it is possible (but unlikely) these are ignored in the prediction. We revise the statement as: "The student’s regularity and video watching behavior were the only two aspects selected as important for the student's prediction of passing the course, and the model did not use any other aspects."
3. A discussion on challenging cases for human-oriented feature grouping, and further clarifications on how these feature groups are used in feature gating and group routing. We discuss this further in the paper in Appendix C. In the difficult case with a feature that belongs equally in two groups, it is better to put a feature in its own subnetwork or combine the groups.

REVIEWER 4 provided a generally positive review, highlighting the “clear and solid motivation of the work” in an “important direction of research”, the “good amount of experiments conducted”, and the “novel” proposed methods) but recommended a marginal rejection. They had several requests motivating this recommendation which have now been fully addressed:

1. A request for a taxonomy and clearer definition of terms in Table 1, and a further discussion of the “faithfulness” and “coverage” of the InterpretCC methods. We have now clarified definitions with connections to prior work, and added a taxonomy of explainability axes in Appendix A, aligning with the terminology in Agarwal et al. 2022, Pinto et al. 2024, and Swamy et al. 2024.
2. A comparison to intrinsic methods that use expert knowledge. There are only a few approaches that do this, mostly focused on the vision domain, where concepts are mostly defined through prototype examples (like SENN). We provide a larger comparison to this work in Sec. 2.
3. A discussion on settings where domain knowledge cannot be easily found. We have now added a nuanced discussion of this for modalities like raw time series.
4. Additional theoretical analysis and intuition of why this MoE approach works. We believe the specialization of subnetworks correlates to the quality of predictions due to the focus on a few input features. Similarly, adaptive and sparse activation contributes to the reduction of noise. We have now added this to Sec. 6.

To summarize the additional experiments, discussion, and revisions (highlighted in blue in the manuscript), we provide the following results to strengthen our paper:

• Additional experiments based on explain-then-predict literature in Sec. 5.1.
• Additional analyses of tuning the sparsity threshold and other parameters, extending the results provided previously in Appendix D.2.
• A full taxonomy of explainability terms in Appendix A, based in prior literature.
• An extended discussion on intrinsic methods with concepts/expert knowledge, the requirement of interpretable basis features, potential bias of explanations, and settings where domain knowledge is hard to identify.
• A theoretical intuition behind the performance of the InterpretCC models in Sec. 6.

Overall, we thank the reviewers for their thoughtful and expert advice that has made us improve the paper much beyond the original submission. We hope that we have adequately addressed reviewer concerns and further improved confidence in our submission.