An Axiomatic Approach to Model-Agnostic Concept Explanations

• It seems that the proposed method is only applicable to datasets with pre-defined concept annotations c(x).
• Though authors show some connections between the proposed method and two existing methods, there still lacks baseline comparisons in the experiment part. For example, why not compare the proposed method with existing methods (such as [cite 1,2]) for model improvement.
• As typical concept explanations usually provide visualizations of extracted concepts. It will help a lot if authors also provide such visualizations, and compare their method with existing concept explanations from this perspective.

[cite 1] This Looks Like That: Deep Learning for Interpretable Image Recognition (NeurIPS 2019)

[cite 2] Concept-level Debugging of Part-Prototype Networks (ICLR 2023)

Some assertions within the paper are inaccurate. Specifically, the authors incorrectly categorize SHAP and LIME as model-specific techniques that necessitate white-box access to the model. Contrary to these claims, SHAP and LIME are model-agnostic and can operate on black-box models. They generate explanations by probing the model with varying input combinations to obtain SHAP and LIME scores.

Furthermore, the terminology used in the problem formulation section lacks precision. The term "measure" is ambiguously defined— it is unclear whether it refers to a "score" or "measurement," or to a "measure" in the context of measure theory. A more mathematically rigorous exposition of the problem setting would be beneficial.

While the abstract and the introduction draw a grand picture of a unified approach that encompasses different existing methods, but ultimately only discussed TCAV and completeness-aware explanations in the paper. The proposed measure is also somewhat similar to how [1] parameterizes a self-explaining neural network (e.g., the product of h(x) and c(x)).

Additional points of concern include: (1) the assertion that the proposed approach subsumes TCAV is not entirely accurate. According to Theorem 4.2, it approximates TCAV in an asymptotic sense rather than subsuming it exactly. (2) The claim that the approach unifies TCAV and completeness-aware explanations is also not entirely precise. Specifically, the class-conditioned variant of the proposed method is linearly correlated with a subset of the completeness score, while its concept-conditioned variant approximates the continuous case of TCAV.

From Section 3, it seems the proposed approach is limited to binary classification. This is a limitation. Would it be generalizable to other cases, e.g., multi-class classification?

Another significant issue that I have is the absence of comparative results with baseline methods. Unless I overlooked it, the paper does not present any performance metrics juxtaposed with at least TCAV and completeness-aware explanations. Such comparative analysis is essential for rigorously assessing the technical and empirical merits of the proposed approach. Without it, evaluating the method's substantive contribution becomes challenging.

[1] Towards Robust Interpretability with Self-Explaining Neural Networks. NeurIPS 2018.

• Presentation issues. The notation is confusing. Concept is denoted with c(x) and is used to denote either set of concepts (i.e. a concept vector) or a scalar score interchangeably causing much confusion, see (1), (2). Yeh et.al. 2020 should be explained in more details in Sec 4.1. h is overloaded to map from example or representations to the label. (2) uses y to enumerate over concept values, which can be confusing.
• Faithfulness is not established. (This is less a weakness and more a show-stopper). The premise of an explanation is its faithfulness, which surprisingly is not one of the axioms. All their estimators at best measure the correlation in activation of concept and the model-to-be-explained. They need to establish the faithfulness of their estimator with the evaluation suite of TCAV or Yeh et.al. to first establish theirs as an explanation. Imagine a model trained on a dataset with two co-occurring concepts but the model may only exploit one of the concepts for prediction (say because one concept is way easier than the other), however none of their estimators can bring out the asymmetric concept importance.
• Unconvincing evaluation. The paper evaluates on a gamble of tasks. The prediction tasks considered are to predict cat, chair, motorbike, bottle in Figure 2, 3, 4 respectively. Some rationale on choice of tasks is expected, and some evaluation using established benchmarks is also expected.
• Computational efficiency not presented. They claim computational efficiency but do not present any empirical evidence supporting the claim.
• Sufficiency vs necessary based score. The paper discusses concept scores based on sufficiency or necessity, but do not discuss their practical implications. When is one better than the other?

1. The experimental evaluation appears to be limited. The analysis in Section 5.2 is limited to a small set of classes, and it is unclear how generally the trends observed hold. The evaluation for model selection also uses very simple models (linear regression and random forest). For broader applicability, it would be useful to evaluate whether this generalizes to deep networks, and whether one can select between different networks and network architectures with this approach.
2. The evaluations in Sectionn 5.2 appear to focus on images that were predicted to be a given class. For different models and different optimizers, this set of images could be different. Given that the ground truth labels of the images do not seem to have been taken into account, it is unclear if the presence of "spurious" concepts observed is due to the model relying on spurious correlations for images where the ground truth class is the same as the predicted class being considered, or due to the model simply misclassifying images of other classes as the predicted class being considered. This would be important to properly evaluate the proposed approach.
3. Given the efficiency and equivalence arguments between the proposed approach and prior work (completeness-aware and TCAV, Section 4), it would be helpful to have an experimental comparison between these approaches in the application settings in Section 5, both in terms of performance and computational efficiency.
4. The formulation of TCAV as an instantiation of the proposed framework seems to only apply to explanations at the last layer, while the TCAV formulation as originally proposed is more general. If this is the case, then this should be made clear.