A Concept-Based Explainability Framework for Large Multimodal Models

Thank you for the positive and insightful comments. Please find our response pointwise below:

Qualitative analysis for feature superposition/specificity of concept vectors: Thanks for the interesting suggestion. We conducted a preliminary qualitative study on some concept vectors in the dictionary learnt for token "Dog" to analyze if these concept vectors tend to activate strongly for a specific semantic concept (monosemantic) or multiple semantic concepts (polysemantic). In particular, we first manually annotated the 160 test samples for "Dog" for four semantic concepts, for which we knew we had concept vectors in our dictionary, namely "Hot dog" (Concept 2, row 1, column 2 in Fig. 7), "Black dog" (Concept 20, row 10, column 2 in Fig. 7), "Brown/orange dog" (Concept 6, row 3, column 2 in Fig. 7), and "Bull dog" (Concept 15, row 8, column 1 in Fig. 7). For a given semantic concept, we call this set $C_{true}$. Then, for its corresponding concept vector $u_k$ we find the set of test samples for which it activates greater than a threshold $\tau$. This threshold was set to half of its maximum activation over test samples. We call this set of samples $C_{top}$. To estimate specificity of the concept vector we compute how many samples in $C_{top}$ lie in the ground-truth set, i.e. $|C_{top} \cap C_{true}|/|C_{top}|$. We find Concept 2 ("Hot dog") to be most monosemantic with 100% specificity. For Concept 20 ("Black dog") too, we found high specificity of 93.3%. For concept 15 ("Bull dog") we observed the lowest specificity of 50%. This concept also activated for test images with toy/stuffed dogs. Interestingly, the multimodal grounding of concept 15 already indicates this superposition with maximum activating samples also containing images of 'toy dogs'. Concept 6 ("Brown/orange dog") is somewhere in between, with 76% specificity. This concept vector also activated sometimes for dark colored dogs, which wasn't apparent from its multimodal grounding.

In summary, your expectation of superposition/polysemanticity is fair. Prominent or distinct semantic concepts seem to be captured by more specific/monosemantic concept vectors, while more sparsely present concepts seem at risk to be superposed resulting in a more polysemantic concept vector capturing them. We can add this discussion in appendix.

Qualifying certain claims: We accept your concerns about the three claims/statements. We will modify them to qualify their extent and scope.

Powerful language model + Generalization to other LMMs: We are pleased to share that we conducted new experiments LLaVA-v1.5. It uses Vicuna-7B (closely related to LLaMA) as the language model. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. The global response presents more details regarding the same.

Absolute activations in Eq. (5): We wrote Eq. (5) this way so that it remains applicable even for methods without the constraint for non-negativity of activations (eg. PCA). For Semi-NMF it indeed does not make a difference.

Trialling gradient-based feature visualisation: This is a fair suggestion and a useful future direction to build upon. For now, we have explored generating saliency maps with LLaVA (not via gradients) for concept activations by computing the inner product of concept vector $u_k$ with all visual token representations from corresponding layer $L$, i.e. $u_k^T [h^1_{(L)}, ..., h^{N_V}_{(L)}]$. This vector of size $N_V =576$ is reshaped to 24 $\times$ 24 and resized to original input. This is feasible for LLaVA as all visual token representations fed to the language model preserve their patch based identity. Sample visualizations are available in 1-page PDF (Fig. 2) in global response.

Typo in Fig. 1: Thanks for pointing it out. We'll correct it.

We thank the reviewer for the interesting feedback and positive comments:

Computational scaling for large number of tokens: Our experiments use single GPU. The BERTScore evaluation does not scale well. It can take upto 3-4 hours to evaluate all baselines on some target tokens (COCO). The representation extraction process scales fine with DePALM (upto 15 minutes for some tokens), but poorly with models like LLaVA (upto 3-5 hours for some tokens). The core of the method itself, dictionary learning and multimodal grounding, is computationally cheap.

Experiments with more tokens: We didn't experiment earlier with more because of expensive BERTScore evaluation. However, we have now conducted experiments with 30 additional COCO classes (apart from 8 in the paper) for CLIPScore and Overlap , with identical hyperparameters. We considered these COCO classes based on the criteria that these are single token words with at least 40 predicted test samples by $f_{LM}$. Single token criterion is to keep our experimental setup consistent. Lower bound criterion on test samples is to ensure average test CLIPScore is reliable for each target token. We report the macro average of test CLIPScores and overlap over the 30 extra target tokens in tables below. We'll add the detailed results in appendix for all baselines. We obtain results consistent with those in main paper, with Semi-NMF extracting concepts with good balance between high CLIPScore and low overlap.

Application to multi-token words: It is an interesting question. None of our evaluated tokens were in this category. However, our approach can be easily adapted to this. In particular, we extract representation of last token from first prediction of the multi-token sequence. The other aspects of the method remain unchanged. While there can also be other viable strategies, the rationale behind this adaptation is that the last token of our sequence of interest can also attend/combine representations from previous tokens in the sequence. We add below results for such examples. Detailed results will be added in appendix):

Motivation to use Semi-NMF: The idea to consider Semi-NMF was inspired from the effectiveness of NMF for various interpretability applications ([1-4]). Given the positive and negative values in $h^p_{(L)}$, it was not possible to effectively apply NMF. Thus, Semi-NMF arose as a natural option by relaxing non-negativity on $\mathbf{U}$. We expected the constraint to only positively combine the dictionary elements should still be useful from an interpretability perspective. Theoretical connections between Semi-NMF being generalized version of K-Means [5] also supported our confidence. We'll expand Sec 3.4 to include some of the motivation for more clarity. We'd also like to add that we did consider all three approaches (PCA, KMeans, Semi-NMF) initially and qualitatively found Semi-NMF dictionaries most balanced (diverse and multimodally meaningful).

Applicability of other current interpretability methods: Related works (Sec. 2) discuss in detail the applicability of previous CAV-based approaches and previous approaches to understand VLMs/LMMs. Within general interpretability literature, the closest related methods to CAV based concept extraction are input autoencoder-based concept methods or concept bottleneck models. However these are used almost exclusively as by-design interpretable networks and thus out of scope for understanding pretrained models. We can add a brief discussion about them in Sec. 2.

Representations extraction + Relation to ROME: We selected representation from first position where target token is predicted following Schwettmann et al. [6] who analyzed neurons for a given caption at the first predicted noun. In practice, what is essential is extracting representations where the target token is predicted. In this regard we are not in conflict with ROME but rather aligned. ROME analyzes factual associations of the form (subject, object, relation). The input prompt to the language model consists of the subject and relation. The model predicts the next token which is expected to be the object and thus ROME also analyzes representation at a position where their token of interest is predicted.

Repeated references: Thanks. We'll correct them.

[1] G. Trigeorgis et al. "A deep semi-nmf model for learning hidden representations." ICML 2014

[2] J. Parekh et al. "Listen to interpret: Post-hoc interpretability for audio networks with nmf." NeurIPS 2022

[3] T. Fel et al. "Craft: Concept recursive activation factorization for explainability." CVPR 2023

[4] YT. Guo et al. "The rise of nonnegative matrix factorization: algorithms and applications." Information Systems 2024.

[5] CHQ. Ding et al. "Convex and semi-nonnegative matrix factorizations." IEEE TPAMI 2008

[6] S. Schwettmann et al. "Multimodal neurons in pretrained text-only transformers." ICCVW 2023

We thank the reviewer for their feedback. We add our response for each point below:

Notation system: We will try our best to reorganize some notations for clarity. We remain open to incorporate any particular suggestions.

Benefits of CAV-based approach for LMMs: Concept based explainability approaches in general are preferred as they highlight what semantic representations a model extracts rather than which regions of input are important for a model [1]. In order to gain understanding about internal representations of an LMM, CAV-type approach were the only viable option for concept extraction as the other types of concept explainability methods (eg. Concept bottleneck models) focus on training interpretable networks by design. Furthermore, the effectiveness of recent works ([2]) in learning concepts dictionaries for CNNs also motivated us to develop a concept extraction approach that can be applied to LMMs.

New experiments: We have conducted new experiments on LLaVA. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. The global response and 1-page pdf present more details regarding the same.

[1] J. Colin, T. Fel, R. Cadène, and T. Serre. "What I cannot predict, I do not understand: A human-centered evaluation framework for explainability methods." NeurIPS 2022.

[2] T. Fel, V. Boutin, L. Béthune, R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. "A holistic approach to unifying automatic concept extraction and concept importance estimation." NeurIPS 2023

We thank the reviewer for their feedback. We address their concerns pointwise below:

Evaluates only on DePALM: We have now also conducted experiments on LLaVA. We are able to extract meaningful multimodal concepts and achieve quantitatively consistent results as for DePALM. Further details regarding the same are available in the global response and 1-page PDF.

Uses two automated metrics, but no human evaluation: Human evaluation would be a good way to further evaluate the performance of our approach. The current paper was rather focused on developing dictionary learning for LMMs, extracting multimodal concepts through dictionary learning, and studying various methods for this goal in a structured way. To the best of our knowledge, the feasibility of these aspects has not been explored before, and was one of the main concerns of the paper. Nevertheless, we would like to consider human evaluation in future developments.

Using CLIP as visual encoder and CLIPScore as metric: We believe the meaningful grounding is influenced more by the language model. We conducted experiments with two non CLIP visual encoders to support this hypothesis. Please find further details in our global response.

Overlap/difference with prior research (Goh et al., Kim et al.) and multimodal neurons literature: We discuss both Goh et al., and TCAV (Kim et al.) in Sec. 2 along with our differences with them. Yes, the notion of multimodal neurons has been introduced previously. However, we respectfully disagree with the assessment that there is a significant overlap with it. Our approach performs dictionary learning on internal representations of a LMM, to extract a set of concepts about a target token. This methodology is quite unrelated to work in (Goh et al.) that analyzes activations of individual neurons in CLIP ResNet encoder (final conv layer) for various input images, to determine what conceptual information in image they activate for. In general, dictionary learning based concept extraction approaches and individual neuron analysis approaches are methodologically different but complementary. They are complementary approaches to analyze internal representations of a network. However, dictionary learning methods discover directions in the representation space useful to decompose data representations, while neuron analysis approaches study activation patterns of individual neurons in detail. We list some more significant differences with Goh et al.:

• The ‘multimodality’ discussed in Goh et al. is different from ‘multimodality’ here or notion of multimodal neurons in Schwettmann et al. In particular, Goh et al. analyze neuron activations only for images where certain conceptual information can be present in different forms/modalities (eg. celebrity as a face in the image, as caricatures in image, as text in the image etc.), while the multimodality here or in Schwettman et al. concerns with explicit grounding of a concept vector or neuron in vision and text.

• The two papers analyze very different types of models (LMMs vs CLIP-ResNet encoder) with different architectures, training objectives and tasks they solve.

Low overlap for PCA: PCA is best for minimizing overlap possibly because its concept vectors by design are orthogonal. Thus, when decoded, they typically yield different sets of grounded words. We would also like to state for clarity that PCA is not learnt for just five concepts, nor do the target tokens ('dog', 'bus', 'cat', 'train') represent concepts of our dictionary. We apply PCA (or any dictionary learning method) to learn a different dictionary of $K=20$ concept vectors for each target token separately. Thus the overlap for dictionary learning methods is reported separately for each token and calculated between 20 concepts learnt for it.

Evidence of polysemanticity: Our preliminary qualitative analysis suggests that concept vectors for prominent and distinct semantic concepts seem to be more monosemantic (eg. 'hot dog', 'black dog'). We also found concept vectors which are more polysemantic in nature, and capture more sparsely present semantic concepts. Such concepts might be identifiable in some cases while extracting the multimodal grounding. For instance, row 8, left column in Fig. 7 illustrates a concept discovered for 'Dog' token that activates for 'bull-dogs', 'pugs' but also for 'toy/stuffed dogs'. More details about the analysis can be found in our comment to Reviewer 17ab on feature superposition.

We thank the reviewer for their positive comments and intriguing questions. We respond to them pointwise below:

CLIPScore and LMM with frozen CLIP encoder: We believe the meaningful grounding is influenced more by the language model. We conducted experiments with two non CLIP visual encoders to support this hypothesis. Please find further details in our global response.

Very recent works on LLM interpretability: Thanks, we make a note of them.

Line 223 typo: Thanks for pointing out the error. We’ll fix it.

Disentangling data effect/model effect: Indeed, the question of the impact of the available data vs the model architecture/training, w.r.t the concepts, is interesting. Qualitatively, in our initial experiments we observed cases where the type of input data clearly affected the quality of extracted concepts. For instance, when extracting concepts for token 'light', we observed most concepts only contained images of 'traffic light', even though we expected more diversity. This was because most input images in our dataset with predicted token 'light' were of 'traffic light'.

On a separate note, it could be worth tracking the evolution of the learnt concepts of our approach during training. This could possibly help to disentangle the training/optimization effect. But we leave that as a future research direction to pursue.

Expanding to more complex concepts: Yes, this is indeed an important direction we are considering to expand in future. One possible way to extend to more complex or abstract words could be to associate the complex word to a set of target tokens. We could then consider decomposing representations for the all tokens in the set simultaneously. Our first instinct is to use a linear decomposition approach. Nevertheless, if this is not sufficient, non-linear decomposition methods could also be explored.