Estimation of Concept Explanations Should be Uncertainty Aware

The proof relies heavily on the statement: "If the examples in D are diversely distributed without any systematic bias, is proportional to the identity matrix, meaning the basis of G and W are effectively the same." Can you justify or prove that? Without the point, the proof is not valid, hence the rest neither.

There may have been some confusion. $A^TA$ is of size $S\times S$ where S is the rank of G. For $A^TA$ to be proportional to an identity matrix, we only require that the examples in the probe-dataset are such that $[g(x_1), g(x_2), …]$ are uniformly spread over the subspace of dimension S. We demonstrate this with an example.

def makeA(N, S):
	return np.array([[np.random.uniform(-1, 1) for s in range(S)] for n in range(N)])
def makeA2(N, S):
	return np.array([[np.random.uniform(-s-1, s+1) for s in range(S)] for n in range(N)])
>>> A = makeA(1000, 2)
>>> A.transpose() @ A
[[334.4,   8.6],
[  8.6, 338.7]])
>>> A = makeA2(1000, 2)
>>> A.transpose() @ A
[[ 341.4,   24.3],
 [  24.3, 1277.0]]

For the second matrix from makeA2, the first dimension is sampled from Unif[-1, 1] while the second from Unif[-2, 2]. Since all the dimensions are not identically distributed, it has systematic biases which prevented $A^TA$ (of makeA2) from being proportional to a diagonal matrix.

We understand that our assumption is not guaranteed to hold since it depends on the diversity of the probe-dataset. Yet, the estimates were found to be empirically very effective as observed implicitly from the high quality of explanations. Moreover, direct evaluation of uncertainty, presented in Appendix G.1, further confirm the empirical effectiveness of U-ACE.

How do you go from $cos(\theta\pm \alpha_k)$ to … How do you know which summant is m(x) and the other is s(x)?

By definition of m(x), s(x), concept activations lie in the range of $m(x)\pm s(x)$. The proof of Proposition 1 finds the range of concept activations as $cos(\theta)cos(\alpha_k)\pm sin(\theta)sin(\alpha_k)$. The result (of $m(x)=cos(\theta)cos(\alpha_k), s(x)=sin(\theta)sin(\alpha_k)$) follows from directly reading off from the estimated range.

What is $u_k$ and $\sigma_k$?

They are the mean and scale respectively of the normal distribution from which the concept activation vectors (w_k) are sampled.

What is a vanilla linear estimator?

We meant ordinary least squares (OLS) by vanilla linear estimator. We corrected the phrase in the latest draft to avoid ambiguity.

"the probability that at least of the K-1 random concepts is estimated to be more important than the relevant concept is ". How do you get this formula?

We modified the text to introduce Corollary 1 and presented the derivation in the proof of Corollary 1. Please see the revised draft.

"the CDF of standard normal", some words are missing here.

We meant $\Phi$ is the Cumulative Distribution Function of standard normal distribution.

"pretrained CLIP model that is publicly available for download." Can you provide a link or reference?

Sure, https://github.com/openai/CLIP/. We also added it in the latest draft.

I understood from Section 3, that U-ACE is built upon a text embedding. How is it learned here?

U-ACE uses text description of a concept to encode a concept through a pretrained multi-modal model such as CLIP. But U-ACE itself does not train any text embeddings.

If Y is computed only for label y of the for-loop, it should be indicated.

Thanks for noticing. We corrected it in the latest draft.

If $v_k$ is defined with respect to a layer $l$, why is there no subscript on $v_k$?

Thanks for your suggestions. We fixed this in the revision.

We thank the reviewer for their valuable time and effort. We are glad that the reviewer appreciates the need to model uncertainty in concept explanations.

The usage of CLIP seems essential, yet it is not involved in the synthetic experiments.

Sorry for any confusion. We would like to point out that U-ACE uses CLIP to encode concepts in all the experiments including synthetic. As described in Section 3.1 and Proposition 1, all our experiments, including synthetic experiments, used text descriptions to encode concepts using the multi-modal CLIP model.

TYPO: Shouldn't it be $\mathcal{D}_c^{(k)}$ in the expected value instead of $\mathcal{D}_c^{(k)}$?

Sorry, looks like your question has a typing mistake, we believe you may be asking $\mathcal{D}_c^{(k)}$ instead of $\mathcal{D}$. If so, we do not see any typo on that line in Algorithm 1. We estimate the expectation over the probe-dataset ($\mathcal{D}$) and not over the concept dataset ($\mathcal{D}_c^{(k)}$). U-ACE assumes $\mathcal{D}_c^{(k)}$ is not available (please see inputs of U-ACE shown in Algorithm 1). $\mathcal{D}_c^{(k)}$ were introduced only to explain how previous concept explanation methods such as TCAV estimate concept activation vectors.

The limitation subsection highlights that such models are very dependent on the probe-dataset, is U-ACE robust to that issue?

Yes, we demonstrated the relative robustness of U-ACE to probe-dataset in Table 2 (“Dataset shift” of Section 4.3) and Figure 3 (“Unreliability due to dataset shift.” in Section 4.1).

Again $W_c\epsilon$ means that either $W_c=0$ or $\epsilon$ is in the kernel of $W_c$. … I infer from the text that the second case does not occur: why?

Sorry, there could be some misunderstanding. We did not assume anything about the kernel or the matrix. We estimated $W_c$ that keeps the value of $W_c\epsilon$ low while also approximating the logits well (please see Alg. 1). U-ACE estimated $W_c=0$ only when all the concepts are irrelevant (please see Appendix F.1 and Proposition 3 for an example). In every other case, the estimated $W_c\neq 0$, which is why we see meaningful explanations such as in Figure 6.

Could you be more precise regarding the order of magnitude of $\lambda$?

If $\lambda$ is very low the constraint $W_c\epsilon\approx 0$ is satisfied but the approximation error can be large. Whereas if the $\lambda$ is very high then the unconstrained regression may lead to better approximation but model is not unaware of uncertainty, i.e. $\|W_c\epsilon\|$ is likely large. $\lambda$ therefore must be neither very small nor very large. We found the best value of $\lambda$ through maximum likelihood estimation (MLE) on training data as described in Appendix B. Therefore the best value of $\lambda$ varied per dataset and was set automatically by MLE.

"prior/posterior on weights": the preposition 'on' is not very clear. In any case, could you give more details about how you obtained Equation 1?

By prior/posterior on weights, we meant $\Pr(\vec{w})$ and $\Pr(\vec{w}\mid \mathcal{D})$ (where $\vec{w}$ according to the notation on page 3 is a row of $W_c$). Please see Slide 10 of https://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=10 for how Equation (1) is derived, note that $m_0=0, S_0^{-1}=\lambda^{-1}\text{diag}(\epsilon\epsilon^T)$, $\Phi=C_X$, Y as defined above Equation 1. We also added the full derivation to Appendix A.1 for the sake of completeness.

What is "cos-sim"?

Thanks for pointing it out, we made it more explicit. We used “cos-sim” as a short form for cosine similarity (https://en.wikipedia.org/wiki/Cosine_similarity).

What exactly is $\alpha_k$

The concept specific value $\alpha_k$ is a measure of uncertainty due to lack of knowledge and due to ambiguity. We added text to Section 3.1 to make the definition clear.

How big is N? It is usually pretty large, so how do you optimize something with let's say 10K dimensions?

N is the number of examples in the probe-dataset and can be very large. But in our experiments, the maximum value of N was only about 10,000 (on the Broden dataset). We pre-computed $e(w_k, g, \mathcal{D})$ and computed $e(v, f, \mathcal{D})$ at every step, and fully loaded both the vectors of dimension N into memory. Thanks for raising this fair point, we may need some special handling for very very large N. We will comment on the same in the revised version.

What is $\theta_k$?

$cos(\theta_k)$ as defined in Proposition 1 is the cosine-similarity between the embedding of the image (x) and the embedding of the concept k ($T_k$).

We are grateful for the reviewer’s thoughtful and passionate feedback. We are very glad that the reviewer found our idea interesting, problem formulation practical, methods compelling/useful and evaluation interesting/convincing.

… could do with a bit of work to more clearly express the main takeaways and experimental evaluations… The presentation of the experimental results could be improved. In 4.1, the "Unreliability due to misspecified concept set" subsection…

We thank the reviewer for their suggestions. We have rewritten the hard-to-parse sections, filled in missing details and presented all the derivations in the Appendix (Please see general response). We also acted on the suggestion made by the reviewer for Section 4.1.

Could you contextualize the significant of proposition 3? …

Thanks for your question. The near-zero importance, i.e. the value of importance score being zero for irrelevant concepts, applies only to the baselines that fit a linear model on the concept activations to estimate the explanation, which is the standard linear estimator baseline considered in Proposition 3. The concept importance score may have a different interpretation for other baselines. For instance in TCAV, any importance score that is not statistically greater than 0.5 is equivalent to zero and is an irrelevant concept. For the ease of analysis, we only considered explanations through linear estimator baselines, which include our two strong baselines: Y-CBM, O-CBM, U-ACE.

Please consider the following response if the question is instead about the need for assigning zero-importance or the need for handling concept sets with no relevant concepts.

Proposition 3 takes the under-complete concept set to the extreme case when all the concepts in the concept set are irrelevant. Appendix F presents an example. In practice, an irrelevant concept set may arise when the human guesses the task-relevant concepts wrong, which is not unusual. Some examples reported in the past are: (a) [1] reported that a standard model (depending on the training data) was found to exploit the green background for recognising cows and yellow background for camels. A user may start cow or camel related concepts and should be able to see they are low importance scores for them all. (b) [2], [3] found that models trained to recognize lung related disease were not even focusing on lung related features. (c) Similar to (b), [4] reported that a model trained to classify skin lesions was found to rely on any lesion related features but instead relied on a special scale that watermarked tumorous lesions.

In all the above scenarios, the user may start with their guess (lung/chest related concepts in (b) and lesion-related concepts for (c)) and should find that the model is not relying on any of the concepts that they considered are relevant.

We are happy to answer any further questions. Thanks again for your careful assessment.

References.

1. Beery, Sara, Grant Van Horn, and Pietro Perona. "Recognition in terra incognita." Proceedings of the European conference on computer vision (ECCV). 2018.
2. DeGrave, Alex J., Joseph D. Janizek, and Su-In Lee. "AI for radiographic COVID-19 detection selects shortcuts over signal." Nature Machine Intelligence 3.7 (2021): 610-619.
3. Zech, John R., et al. "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study." PLoS medicine 15.11 (2018): e1002683.
4. Narla, Akhila, et al. "Automated classification of skin lesions: from pixels to practice." Journal of Investigative Dermatology 138.10 (2018): 2108-2110.

…It doesn’t make sense to ask what the importance of green is when there are no green images in the probe set (Fig3 middle) or what the importance of red is when the images are all green. … a practicioner should notice that their probe set contains no example of green before ….

Thanks for your question. We agree that choosing probe-datasets carefully can solve the problem partially as was also suggested by Ramaswamy et.al.: “The choice of the probe dataset has a profound impact on explanations…. Hence, probe datasets should be chosen with caution.” However, when the number of labels, concepts and examples in the probe dataset are in the order of thousands, manually checking for presence of each concept with each label over all the examples can be impractical. Which is why we believe (among other reasons) that a concept explanation must be equipped with confidence scores on their estimate of importance, which insulates from misinterpreting importance scores of missing concepts. Our simulated study with the colored dataset is inspired by the difficulty in carefully picking the probe-dataset or concept set when operating at scale.

…because if the concept vectors are directions in the model latent space, then one would think that a concept attribution explanation is explainable merely by computing the projection of the model’s linear layer with the concept vector… learn a new linear model to estimate concept importance.

True that “then one would think that a concept attribution explanation is explainable merely by computing the projection of the model’s linear layer with the concept vector”, but such a point estimate is bound to be noisy. m(x) is simply the mean over many such projections (using several samples of concept activation vector).

So what is the source of uncertainty that U-ACE accounts for?... why can’t TCAV be equipped with a simple data uncertainty measurement? …

We answered the sources of uncertainty in our response above.

TCAV is already equipped with a simple uncertainty measurement to distinguish a truly important concept from a random concept. TCAV computes m(x) and s(x) of concept activations by simply training multiple concept activation vectors. Yet, TCAV estimated explanations are noisy as seen in Table 1 likely because simple measurement of uncertainty through MC sampling may fail to capture epistemic uncertainty as can be observed from the results of Appendix G.1.

“Simple estimates explanation using concept annotations and therefore its explanation must be the closest to the ground-truth” — What would the ground-truth look like for the experiment in 4.3?

We noticed that concept annotations do not mark all the concepts present in an example, which is understandable given the number of concepts: 720. Since “Simple” is trained on ground-truth concept annotations, it may miss attributing importance to the concepts that are systematically missed by an annotator. We expect therefore that the most salient concepts identified by “Simple” are a subset of truly salient concepts. Nevertheless, “Simple” trained on concept annotations is the closest explanation to ground-truth, which we use to evaluate other methods.

typo: UNCERTAINITY typo: while other methods the importance

Thank you, we fixed these in the revision.

“Since the co-occurrence probability of U with car class goes from 1, 0.5 to 0, we expect the importance score of U should change from positive to negative as we move right” — It doesn’t change to negative in the graph, although it does get lower

Thanks for bringing it up. The importance score does go negative but since we visualized in Figure 5 the rank-normalised concept importance scores (as explained on page 6), it can only be positive. We fixed the text accordingly.

We are happy to engage in further discussion to clarify any unresolved concerns. Thanks again for your detailed review.

We thank the reviewer for their passionate and thoughtful assessment. We are glad that the reviewer found our motivation and contributions clear, and our evaluation convincing. We address their concerns below and are very happy to engage further.

This paper attempts to address two topics at once, and its novelty is severely undercut by existing work in both directions. … Anyway, the paper comes on the heels of a number of works on unsupervised concept detection, including… However, … this paper uses … CLIP, ... not novelty ….

Sorry, there may have been some misinterpretation. We would like to clarify that U-ACE assumes that the concepts are supplied and we did not propose a method for concept discovery (although it is an important problem in itself). As explained in Section 2, the list of concepts $\mathcal{C}=\{c_1, c_2, \dots, c_K\}$ are supplied along with their text descriptions: $\{T_1, T_2,\dots, T_K\}$.

On the concept estimation side, there is older work… what is the difficulty in uncertainty for the vector v_k in Sec. 3.1, which is the concept vector, using any number of standard uncertainty measurements (whether frequentist or Bayesian)?

Thanks for raising this issue. To further motivate the need and effectiveness of uncertainty estimation of U-ACE (described in Section 3.1), we contrasted U-ACE with uncertainty estimated using MC sampling or distributional parameter fitting (similar to ProbCBM as the reviewer suggested). The results can be found in Appendix G.1 of the revised draft. We find that U-ACE estimated uncertainty is high-quality as well as compute-efficient. Thanks for pointing to Slack et.al. work, although their methods are not trivially applicable to our problem we will mention the paper in the related work and highlight the common intent.

…How is s(x) computed? What is sin(x)? … could not tell where the uncertainty was supposed to come from.

sin(.) is in fact the sine function, we will make it explicit. Two major sources of uncertainty in concept activations that are modeled by U-ACE are: (1) epistemic uncertainty arising from lack of information about the concept in the representation layer of the model-to-be-explained, (2) data uncertainty arising from ambiguity likely because the concept is not clearly visible (see Appendix G.1 for some examples). We demonstrate the merits of U-ACE in accounting both the uncertainties in Appendix G.1. We also added passages to Section 3 to clarify the sources of uncertainty.

… sparsity-encouraging prior would have been more appropriate. But the paper aims to encourage weights to be orthogonal to a noise vector. What is this noise vector? … I have no idea why this would be a good prior for the model.

Sorry for any confusion. We have two motives for the linear estimator $W_c$ (which is returned as the explanation): (1) should be robust to error in concept activations, or in other words, the linear estimator should not rely on concepts with large noise/uncertainty (3rd para of Section 3), (2) should be sparse for easy interpretation. We enforce (1) by imposing a prior such that the samples are almost orthogonal to the noise vector. Please see Equation 1 and text above it for more details about the noise vector ($\epsilon$: a vector of size K: the number of concepts) and justification on why it is a good prior for enforcing the requirement (1) above. Our prior is not designed to enforce sparsity (2); we sparsify weights post-hoc as explained towards the end of Section 3 and before Section 3.1. Further in Appendix G.2, we evaluated the role of prior in producing quality explanations.

Fair comparison with baselines: How was Y-CBM tuned, and what exactly is the final different with U-ACE? … Is the technical contribution here mainly the prior? How would an L1 or another sparsity inducing prior affect Y-CBM performance in Table 1?

Major difference between Y-CBM and U-ACE is that U-ACE estimates the linear weights using Bayesian regression with prior as described in Section 3 while Y-CBM estimates the same using lasso regression. Since the prior on U-ACE’s linear weights is a function of uncertainty in concept activations, it is aware of uncertainty.

Estimation of explanations using Bayesian regression with an uncertainty-aware prior is one the crucial contributions of the paper. We wish to highlight that the contribution of our work also lies in demonstrating the role of uncertainty in concept explanations, and in estimation of uncertainty in concept activations.

To address your concerns more concretely, we did the following: (a) Appendix G.3 presents results for Y-CBM and O-CBM for varying values of the regularization strength, and still perform worse than U-ACE (without any hyperparam tuning), (b) Appendix G.2 contains comparison with Bayesian regression without the uncertainty aware prior (of U-ACE). The experiments paint a clear importance of the U-ACE’s prior.

We are grateful for the reviewer’s thoughtful feedback. We are very glad that the reviewer found our problem practical, motivation convincing, and evaluation setup strong.

Implications in Section 3: … Why does a high probability bound on the dot product of the weight and noise imply that? What is the distributional assumption on the noise? Also, can the authors explain how they arrive at eqn 1?

Thanks for pointing out these issues. We added more detail in the main paper, added detailed derivations in the appendix to improve the readability of Section 3. We are happy to resolve any unresolved concerns.

We did not make distributional assumption on the noise and fully derived the Eqn (1) in Appendix A.1.

Sparsifying the weights: Instead of picking the threshold by hand, why can't the authors impose a lasso penalty here as is done for the simple baseline?

Depending on the choice of prior, we may not get a clean analytical form for the posterior on the weights as shown in Equation (1). Although our post-hoc sparsification of weights is not fully Bayesian and introduced an additional hyperparam, it worked satisfactorily for all our experiments.

Section 3.1 is confusing: … just call CLIP/Multimodal function an embedding function.

We agree. We have revised the draft to emphasize the generality of our method to any multi-modal model.

…I … don't understand how the mean and errors are computed….

We greatly appreciate your concern. In Section 3.1 of the revised draft, we have added intuition and filled in the missing details. Could you please take a look and let us know?

Is there a way to abstract away the specific form of the concept activation of Oikarinen et al. (2023) in this formulation?

Thanks for the great observation, we agree. The role of Oikarinen et al. (2023) is only to inform us about how much information about the kth concept is encoded in the representation layer of the model-to-be-explained, which we denote by $cos(\alpha_k)$. We can use any method for estimation of $v_k$, we just used Oikarinen et al. (2023). In the revised draft, we changed the writing to abstract out specific form of estimation: Oikarinen et al. (2023)

The problem with CLIP: The formulation here ties in with CLIP intimately. …

As the reviewer pointed out in their previous concern. U-ACE has very little to do with how and where the image and text embeddings: $g, g_{text}$ come from. Although our evaluation setup used CLIP, any other pretrained multi-modal model can be plugged-in very easily. We understand that CLIP is not suitable in biomedical applications, but we can simply replace CLIP with an appropriately pretrained multi-modal model.

Why focus on the post-hoc CBM setting?

We request the reviewer to please rephrase or elaborate? We did not understand the concern. We have concept annotation on full training data for all our settings. However, only TCAV and Simple have access to them. Y-CBM, O-CBM and U-ACE did not use the available concept annotations in any way.

Regarding "chance to at least demonstrate the importance of the uncertainty estimation part of their work.", Appendix G.1 of the revised draft demonstrates the quality of uncertainty estimated by U-ACE.

We thank the reviewer once again for their thorough assessment and are more than happy to engage further.

We are grateful for the reviewer’s time and efforts. We are very glad that the reviewer found uncertainty modelling as novel and important, problem well-motivated and topical, evaluation setup well aligned with the key message, and our results convincing.

Clarity: The text is definitely too dense in parts (especially Section 3). Some paragraphs feel rushed and would enjoy a rewrite. There are also some typos (I mentioned a few below). Minor issues…

Thanks for pointing out these issues, all the minor issues are now resolved in the revised draft. We have also rewritten parts of Section 2, 3 to improve clarity and readability.

It wouldn't hurt to clarify the steps in the mathematical derivation. Also, why is s(x) being averaged over? Why can't it be used as-is?

We did it for the ease of analysis. We get a clean closed form if we average over s(x).

If is the inverse variance of noise in observations, why is it being optimized?

Thanks for raising this issue. We added the following passage to the paper. We could directly set the inverse of $\beta$ approximately 0 since there is no noise on the observations: Y. Instead of setting $\beta$ to an arbitrary large value, we observed better explanations when we allowed the tuning algorithm to find a value of $\beta, \lambda$ to balance the evidence and noise.

Section 3 should be unpacked to facilitate understanding.

Thank you very much for pointing out the problems. We have rewritten Section 3 to improve its readability.

… I am under the impression the implementation is model specific. Could you please confirm this? If so, how should m(x) and s(x) be estimated for other models?

Our implementation is not specific to any model. We only assume access to the representation layer and the logits of the model-to-be-explained. We did not make any specific assumptions about the multi-modal model. CLIP can be easily replaced with any other pretrained multi-modal model in a plug-and-play manner.

A pretrained multi-modal model defines g, $g_{text}$ of Section 3.1 and any new model-to-be-explained must define ‘f’ (a mapping from the input to the representation and logit). The rest of the details must follow for computation of m(x), s(x) as described in Proposition 1.

We have revised the draft to emphasize the generality of our proposal to any (neural-network-based) model and any multi-modal model.

Please let us know if we have misinterpreted your question or you found our response not convincing.

We thank the reviewer once again for encouraging feedback and careful assessment.

We thank the reviewer for their response.

Both the remarks are addressed in the (just) revised manuscript. Thanks for the great catch! We agree that we used "noun-colon-symbol" overly in the previous draft.

We are more than happy to answer any unresolved concerns. We thank the reviewer once again for engaging in discussion with us.

Gentle reminder that the author-discussion deadline is in few hours. We are happy to answer any unresolved concerns. Please consider championing our paper if you like it.

We thank the reviewers once again for immensely useful feedback/discussion so far.