Testing Semantic Importance via Betting

Thank you for your comments!

We thank the reviewer for their encouraging comments. Here, we address each weakness point individually, and we are looking forward to discussing more.

if really works on input features from user, is a very important contribution.

We do want to stress that our method does work with any set of concepts, and in particular it is the only method that can provide local semantic explanations for the predictions of a black-box model. So, we do appreciate the reviewer's comment that this is a very important contribution!

Clarity of presentation

Could the reviewer expand on their points of confusion? We would be more than happy to clarify methods and equations in the revised version of the manuscript.

Alignment with human intuition

With agree with the reviewer that a user study needs to be performed to claim alignment with human intuition. As we state in the general response at the top of our rebuttal, we will smooth such claims in the revised version of the manuscript. We note that our contributions can readily work with any set of concepts, including user-defined ones, and that no alternatives currently exist.

We have included several additional experiments to strengthen the experimental section of our submission and validate the ranks of importance obtained with our methods.

In particular, we have included the AwA2 and CUB datasets, which have ground-truth annotations of which concepts are present in images coming from certain classes or specific images, respectively. We refer the reviewer to our general response at the top of our rebuttal for a detailed description of the additional experiments and their results.

To summarize our findings:

• c-SKIT has better semantic importance detection performance compared to PCBM on AwA2.
• x-SKIT has good alignment with ground-truth annotations across all models ($\approx 0.85$ average $f_1$ score) on CUB.

Thank you for your comments and questions!

We now address each point raised by the reviewer individually, and we are looking forward to clarifying any outstanding questions.

Diverse datasets and models

We thank the reviewer for their suggestions, which have significantly strengthen our experimental results and uncovered interesting findings.

As presented in the general response, we have now included additional experiments on both AwA2 and CUB-200-2011, comparing across 8 different models: CLIP:RN50, CLIP:ViT-B/32, CLIP:ViT-L/14, OpenClip:ViT-B-32, OpenClip:ViT-L-14, FLAVA, ALIGN, and BLIP.

In particular:

• Since AwA2 includes class-level annotations, we have included global conditional importance results with c-SKIT in comparison with PCBM.
• Since CUB includes image-level annotations, we have included local conditional importance results with x-SKIT.

We describe all findings and experiments in the general response at the top of our rebuttal. To summarize, we find our c-SKIT outperforms PCBM both in terms of semantic importance detection and transferability across all models on both Imagenette and AwA2. Furthermore, x-SKIT importance ranks align well with ground-truth annotations ($\approx 0.85$ average $f_1$ score) on CUB.

We consider NLP and language generation applications of our framework as important future directions. For example, it is still not completely clear how ideas of concept bottleneck models apply to encoders such as BERT. How do text encoders represent semantic information with standard pretraining techniques such as masked language modeling? Only very recent work [1] has started addressing these fundamental questions. Going beyond text encoders, how should one phrase questions of semantic importance for contrastive encoder-decoder architectures such as CoCa [2] or autoregressive models such as GPT? The structured nature of these models raises very important questions that currently remain unanswered, thank you for raising this point.

We will include these discussion points in the revised version of the manuscript.

User study to evaluate alignment with human intuition

As stated in the general response, we agree with the reviewer that a user study would be necessary to claim alignment with human intuition, and we will rephrase those claims in the revised version of the manuscript. However, we remark that our tests can be readily applied to any set of concepts, including user-defined ones.

We believe the design of a robust user study to deserve its own investigation, and, in this submission, we focus on introducing the methods that will enable such a study. We stress that, currently, no alternative method can work with a few user-defined concepts. For this reason, we envision our method to enable the design of such studies.

Validation of concept usefulness

We thank the reviewer for their suggestions.

We validate important concepts in our additional experiments on AwA2 and CUB using the ground-truth annotations. We refer the reviewer to our general response and the attached pdf for numerical results and comparison with PCBM.

Furthermore, we would like to mention that our c-SKIT and x-SKIT tests precisely work by resampling concepts and measuring their effect on distributions of the output of the predictor. However, we remark that we use observational conditional distributions (i.e., $Z_j \mid Z_{-j}$) and not interventional distributions (i.e., $Z_j \mid \text{do}(Z_{-j} = \cdot)$). Characterizing the connections between our framework and causal inference is an interesting future line of research, thank you for raising this point.

Lastly, we kindly push back on the suggestion of validating concepts by training predictors, as our focus is to study which concepts are important for a fixed black-box model, and, in the general case, these may be different from the ones with highest predictive ability. This is a fundamental distinction between our approach and alternatives like PCBM. Intuitively, consider ImageNet classification. It might be the case that we want to test one concept only. But of course, even if that individual concept were truly used by the model, it will not be enough to train a good classifier from scratch. We can already see this behavior in the AwA2 experiments: restricting to 10 concepts, that we know from ground-truth annotations should be important, significantly reduces predictive power. We refer the reviewer to the "average" line in Table 1 of the attached pdf, where PCBM shows a drop in performance of around 4%.

[1] Tan et al. "Interpreting pretrained language models via concept bottlenecks." (2024)

[2] Yu et al. "Coca: Contrastive captioners are image-text foundation models." (2022)

Thank you for your comments and questions!

Here, we address each point individually, and we are looking forward to discussing with the reviewer.

Limited experiments and transferability across different vision-language models

This is a great question, and we thank the reviewer for raising this point. These additional experiments have strengthen the experimental results of our submission, and uncovered interesting aspects of transferability.

In particular, we have extended our experimental results to include 2 additional datasets (AwA2 and CUB) and 8 different models (both CLIP- and non-CLIP-based): CLIP:RN50, CLIP:ViT-B/32, CLIP:ViT-L/14, OpenClip:ViT-B-32, OpenClip:ViT-L-14, FLAVA, ALIGN, and BLIP.

We evaluate agreement between pairs of models in terms of ranking of concepts, and whether they are classified as important or not. To compare ranks, we use a weighted version of Kendall's tau (see [1]) which assigns higher penalties to swaps at higher positions. That is, for example, a 1 -> 5 swap is worse than a 4 -> 5 swap. This reflects that higher positions should matter more and be more stable. To compare importance, we threshold rejection rates at level $\alpha$ and compute the accuracy between the binarized vectors.

We briefly summarize here the findings of our experiments, which are described in the general response:

• Ranks obtained with c-SKIT are more transferable than PCBM on both Imagenette and AwA2.
• Both CLIP- and non-CLIP-based models are generally aligned in terms of ranks and importance, especially on local explanations.

Compositional understanding of CLIP's embedding space

This is a very interesting point, which deserves further investigation outside of the current submission.

For example, following the SugarCrepe example of a photo with "a girl in white facing a man in black", one could test whether the prediction of "girl" does depend on the concepts "white", "facing"; while the prediction of "man" depends on "black". We stress that this study would not be possible without the tools presented in this submission.

We would also note that our results on global (marginal) importance find that concepts are almost always important (i.e., their rejection rates are above $\alpha$). As suggested by the reviewer, this finding may support the claim that CLIP's semantic space is entangled and overlapping. These aspects have also been considered by previous works (for example, MERU [2]), which focus on whether using an embedding spaces different from the unit sphere (e.g., the hyperbole) induce better hierarchical representations. In a similar fashion to above, one could also devise an experiment to retrieve and test semantic image traversal as in [2, Figure 5].

These example studies highlight that the framework presented in this submission has a broad reach beyond explainability and it will support other research efforts, thank you for raising this point.

References

[1] Vigna. "A weighted correlation index for rankings with ties." (2015)

[2] Desai et al. "Hyperbolic image-text representations." (2023)

Thank you for your comments and questions!

We address each point individually, and we are looking forward to discussing with the reviewer to answer any outstanding questions.

Comparison with SOTA

We thank the reviewer for this comment, which has significantly strengthened our experimental results. We have described several additional experiments and comparisons with PCBM (which is the state-of-the-art for semantic explanations) in the general rebuttal. To summarize, we find that:

• c-SKIT provides ranks that are more transferable across different vision-language models on both Imagenette and AwA2.
• c-SKIT provides better semantic importance detection in terms of $f_1$ score on AwA2.

For local semantic explanations, we have included additional experiments on CUB-200-2011, which indicate x-SKIT is well-aligned with ground-truth ($\approx 0.85$ average $f_1$ score). We note that, currently, there are no alternative methods that can produce local semantic explanations, hence why we could not compare.

For the sake of completeness, for global conditional importance, we tried comparing with LaBo (Yang et al., 2023), which, intuitively, adds a softmax activation to the weights of a PCBM classifier. This approach, however, fails to learn good predictors in our few-concepts setting ($\approx 20\\%$ classification accuracy on AwA2 dataset compared to $\approx 95\\%$ for PCBM).

Small number of concepts

We are not sure we fully understand the question. Could the reviewer expand on what they mean by ``enough''?

Our proposed framework does not modify the original black-box predictor, whose accuracy does not depend on the number of concepts. So, defining a small set of concepts does not affect the performance of the predictor.

On the other hand, it is true that the number of concepts may thwart the ability to build effective samplers to instantiate our c-SKIT and x-SKIT tests. For example, in the case of limited data, it may not be feasible to train a conditional sampler on a large set of concepts. In our experiments, we focus on $\approx 20$ concepts because previous work [1] has shown that humans prefer succinct explanations. We found that this assumption allows us to use non-parametric samplers which are fast, cheap, and do not require prior training.

We will clarify this in the revised version of the manuscript, thank you.

What are the computational limits? How do the tests scale with increasing data size and complexity?

We thank the reviewer for these questions.

First, the main computational limit is the need of conditional samplers. In certain domains, these models may be expensive both to train and run, such is the case for diffusion or language models. In our experiments, we strived to use methods that are effective but do not require prohibitive computational resources.

Second, the computational complexity depends on the specific test and the sampler. In particular:

• SKIT: Following [2, Appendix F.2] the test runs in $O(\tau^2)$, where $\tau$ is the (random) stopping time of the test. This is because at each step $t$, computing the MMD requires summing over the previous $t-1$ terms.
• c-SKIT and x-SKIT: both tests incur in a extra factor of $T_n$, which represents the cost of the sampler on $n$ data points.

Finally, we note that the runtime does not depend on the number of concepts because different concepts can be tested simultaneously. We will include this in the revised version of the paper.

For what data modalities is the given choice of kernel functional?

This is a great point!

Our framework assumes the black-box predictor can be divided into an encoder and a classifier, and these need to be appropriate for the data modality at hand. Once inputs are mapped to an embedding space, one needs to decide on which kernel to use to test for semantic importance.

In general, when using kernel methods to test for a null hypothesis, we would like the kernel to be characteristic for the alternative (see [2]). That is, we want the kernel to be expressive enough to distinguish two distributions coming from the alternative. Over compact domains, this can be achieved by using universal (see [3]) kernels---such as the RBF kernel.

In practice, this means that:

• for x-SKIT, the RBF kernel is appropriate whenever the classifier is a real-valued function (e.g., linear classifier with sigmoid activation). For discrete predictors (e.g., decision trees), other kernels may be necessary.
• for c-SKIT and SKIT, the RBF kernel is appropriate whenever both the predictor and the concept bottleneck layer are real-valued functions (e.g., linear classifier with sigmoid activation and SVMs, respectively).

We remark that our tests are defined for any choice of kernel, and they can be instantiated directly for the desired data modality.

Finally, we refer the reviewer to Fig. E.3 in the Appendix, where we precisely compare c-SKIT with a linear and RBF kernel on a synthetic dataset. This experiment shows that a linear kernel may fail to detect important concepts because it does not satisfy the universality property.

References

[1] Ramaswamy et al. "Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability." (2023)

[2] Podkopaev et al. "Sequential Kernelized Independence Testing." (2023)

[3] Sriperumbudur et al. "Universality, Characteristic Kernels and RKHS Embedding of Measures." (2011)