Thank you for your comments! We appreciate your feedback and address your concerns below.

Sparsity/interpretability-accuracy tradeoff. We thank the reviewer for their detailed thoughts on the interpretability-accuracy tradeoff claims and for their willingness to increase their score if this issue is resolved. The reviewer is correct that Figure 3 highlights the existence of tradeoffs between interpretability and accuracy, and we will correct the language around these claims and address this limitation in the final version. However, we note that this tradeoff is small at desirable sparsity levels. For a sparsity level of 20-30 (chosen with respect to a prior human evaluation study [49]), we find a drop in zero-shot accuracy of less than 4% for the CIFAR100 dataset (Fig. 3). Furthermore, this drop is entirely recovered when classification is done via probing instead of zero-shot (please see Appendix C.1). Secondly, this tradeoff is not necessarily fundamental, as it is possible to retain downstream performance by using SPLiCE as a post-hoc concept explanation method. More specifically, we can add the residual error back into the reconstructed embedding to recreate the CLIP embedding before use in downstream applications (as suggested by reviewer Vxod). Thus, we can achieve interpretability without affecting performance. We do note that in this case, the residual will still remain uninterpretable, which is why we do not focus on this setting in the main paper. We will be sure to discuss this in more depth in the final version, including the key limitations of both of these settings.

As requested by the reviewer, we included experiments on four additional datasets from the CLIP paper [1] (Caltech-101, SUN397, STL10, VOC 2007) in the additional results, Table 1, for sparsities of 20-35. While there is a decrease in performance, we believe it to be relatively small and recoverable by probing or the technique discussed above.

Technical contributions and exploration of alternative existing solutions. Thank you for this interesting question. We note that we explored two alternative approaches in the main paper, including an alternative solver, ADMM, and a method for learning the dictionary, FISTA, but we will elaborate on this discussion further in the final version. In Appendix B.2, we discuss the use of Alternating Direction Method of Multipliers (ADMM) in place of scikit-learn's Lasso method to solve the SPLiCE objective with GPU/batched speedup. We present results in Figure 2 of the additional material, illustrating the equivalence of these methods in both cosine reconstruction and zero-shot accuracy for CIFAR100.

We also explore an alternative dictionary learning method, similar to those suggested by the reviewer (Sparse NMF, KSVD, Sparse Autoencoders, PGD), of sparse dictionary learning with nonnegative weight projection via Fast Iterative Shrinkage-Thresholding (FISTA) in Figures 3, 7. We note that FISTA and the other suggested methods all learn uninterpretable dictionaries that require post-hoc human labeling of the learned concept dictionary atoms. While existing literature such as [31] explores methods for visualizing concepts learned by NMF, PCA, k-Means, and Sparse Autoencoders, this process still requires manual labeling of concepts, which can be both data- and time-intensive. One of the key benefits of SpLiCE is that we fix the concept atoms a priori rather than having to analyze and label components post-hoc. Our results also demonstrate that a learned dictionary results in better reconstruction in terms of cosine similarity, as expected, but surprisingly underperforms our LAION dictionary on zero-shot classification, due to the lack of semantic structure within these learned dictionaries.

Mathematizing. Thank you for the comment. We plan to adopt your feedback, move the empirical validation found in Appendix B.5 to the main paper, and reduce the length of our formal analysis, deferring our theoretical results to the appendix. For further discussion, please see our general comment.

Vocabulary choice. This is an important point we intend to discuss in the final paper. While we considered WordNet, we preferred LAION as our source of concepts because it is the training dataset of Open-CLIP (the model used in this work), preventing us from including concepts in the dictionary that CLIP may not have learnt. Please see our general comment for a more thorough discussion of this point.

Mean centering. We did consider Frechet means, but we found it non-trivial to define a clear notion of mean-centering on the hypersphere. Because the surface of a unit sphere does not contain the “zero” element necessary to define an algebraic field, it is hard to define addition or subtraction on the sphere and, subsequently, mean-centering. We found that Euclidean mean-centering sufficiently closed the modality gap for our purposes; however, if the reviewer has specific suggestions regarding this, we would be happy to update our centering method.

Zero shot in the limit. We do expect to recover CLIP’s baseline performance in the limit when our sparsity constraint allows for a 512-dimensional solution, aside from shrinkage incurred by the L1 penalty. In our experiments, we find that sklearn’s Lasso algorithm does not converge with minimal L1 regularization, so we solve with Ridge regression and truncate to the top 512 coefficients. Table 2 in the additional material shows that at 512 coefficients we are indeed able to recover CLIP’s baseline performance. Note that this still provides interpretable decompositions and so is directly preferable over CLIP.

Limitations. We will be sure to move our limitations section up to the main body of the paper for our final submission and will elaborate on the accuracy-interpretability tradeoff you have mentioned above.

We again thank you for your feedback and hope these comments will encourage you to reconsider your score.

Thank you for your comments! We appreciate your feedback and address your concerns below.

Benefits of sparse CLIP embeddings. Thank you for highlighting this confusion, we hope to clarify this in our response. The primary benefit of SpLiCE is the insight it provides into interpreting the semantic content of the underlying images encoded by ordinary CLIP embeddings. We show in the paper (in Section 6) that this additional insight opens up at least two applications: detecting biases in datasets and intervention on the concepts encoded, both of which are not possible with ordinary CLIP representations. SpLiCE provides a computationally efficient method for understanding the semantics of individual images (See Fig. 2 and additional results Fig 3) and for understanding the semantics of unstructured datasets, allowing for the exploration and summarization of data even without labels. We answer the specific questions that the reviewer has about the two applications below.

OOD evaluation setting. Thank you for bringing up this important evaluation setting. We kindly note that we already consider the OOD setting in the Appendix C.5 with the Waterbirds dataset, which has a spurious correlation in its train set and a distribution shift in its test set. SpLiCE allows us to identify distribution shifts in two ways: first, we can evaluate the similarity in distributions between decompositions of test images and validation images (as we did in Table 6 of the Appendix for Waterbirds), or we can assess the weights learned by a linear probe to better understand how correlations in the train and test data might differ. For example, we see that we can detect the spurious correlation of landbirds and land backgrounds in our probe trained on SpLiCE weights, and when intervening on the highly-weighted land-related concepts [“bamboo”, “forest”, “hiking”, “rainforest”, and any bigrams containing “forest”] by setting them to zero in the weights of our probe, we are able to increase accuracy for the subgroup “waterbirds on land” in the test distribution (Appendix Table 5). In other words, SpLiCE’s interpretable sparse decomposition admits the identification of OOD spurious correlations (e.g., landbirds and land backgrounds), and also provides an interpretable strategy for intervention by allowing for fine-grained control via semantic concepts.

Significance of Woman-Swimwear bias. Thank you for the question. This study shows that within the “swimwear” attribute, women are significantly overrepresented when compared to men. Another way to view this is that 10% of the images of women in CIFAR100 depict them in swimwear (generally referring to bikinis) or underwear. We believe this is a serious problem of bias in the dataset, and constitutes a “representational harm,” in that these stereotypes present in the dataset will be propagated with the use of this dataset to downstream tasks. This can result in downstream risks of bias such as 1) a generative model learns that when prompted for photos of women, it should generate this woman in revealing clothing or underwear, or 2) an image retrieval or generation algorithm failing to be representative when queried for “A photo of a CEO” by returning primarily men due to women being less likely to be depicted in corporate attire. While the reviewer points out the lack of downstream performance evaluation and improvement, we highlight that the goal of our work and this case study is not to develop a bias intervention but rather to propose an interpretability method and demonstrate its usefulness, and that this bias is a property of the dataset itself, which can result in different impacts for different models and tasks.

Human evaluations. We agree that user studies are an essential part of evaluating human interpretability, and we have thus included results for a small-scale human evaluation based on the user study presented in [23] in our additional results (Figure 1). We evaluate our method along with two baselines [22, 23] to assess how relevant the concept decompositions are to the input images, how relevant the concept decompositions are to the model’s prediction, and how informative the decompositions are. We find that users significantly prefer our method to the baselines for both relevance to the input images as well as informativeness, further validating the interpretability of our method. We direct the reviewer to our general comment for more information about the study setup and results.

Reliance on set list of concepts. Thank you for this comment. We agree that spurious correlation and bias detection are dependent on the spurious features being present in the concept vocabulary, thus posing a limitation. We note that we attempt to make our vocabulary as general as possible by constructing it from LAION, the training dataset of Open-CLIP, however it is true that this set may still not include the desired concepts for some downstream applications. In addition, we note that we find this vocabulary to be empirically sufficient for a variety of tasks (For more tasks see additional results). However, we will be sure to acknowledge this limitation in the paper and make this point more clear in the final version. For a more detailed discussion on our constructed vocabulary, please see our general comment.

Practicality of Assumptions. This is an insightful question. Our empirical evaluations all strive to validate the assumptions proposed in section 3. In addition, we include in Appendix B.5 an additional sanity check of CLIP’s linearity (Assumption 3). We note that we intend to de-emphasize these assumptions in the final version of the paper in favor of our empirical validation from Appendix B.5, as suggested by reviewer djnP. For further discussion of these assumptions, please refer to our general comment.

We again thank you for your feedback and hope these comments will encourage you to reconsider your score.

Dear Reviewer nCHL,

As the author-reviewer discussion period is about to close, I would like to know your thoughts on the author rebuttal. Especially since the other reviewers all currently leans towards acceptance, it would be extremely informative for me to know if you still maintain your original score following the rebuttal. I would very much appreciate if you could reply to the authors before the close of the discussion (Nov 13 11:59 pm AoE).

Gratefully,

AC

Thank you for your comments! We appreciate your feedback and address your concerns below.

Connection to Multi Modal Topic Models. This is an interesting connection! MMTMs such as [A] are trained on a corpus of data and use tf-idf statistics to generate multimodal clusters of topics, such as clustering images with text topics. However, these models are designed to explain datasets and not individual samples like SpLiCE. Additionally, this means every image in a cluster gets the exact same image tag and score, which prevents any comparison or differentiation between samples in a dataset despite these images being unique. Furthermore, we could use SpLiCE as a MMTM by generating concept decompositions for each image and applying simple k-means clustering on our interpretable decompositions. Finally, we note that the goals of this work are not to explain a dataset but rather to study CLIP and leverage it to explain individual image embeddings. In doing so we unlock new use cases, one of which is the capability to explain a dataset. We thank you for pointing out this connection, and we will be sure to discuss this work in the final version of our paper.

Manual evaluation experiments. We agree that human evaluation studies are important for assessing the human interpretability of concept based explanations. We have included results from a small-scale human evaluation based on the user study presented in [22] in our additional results (Figure 1). We evaluate our method along with two baselines [22, 23] to assess how relevant the concept decompositions are to the input images, how relevant the concept decompositions are to the model’s prediction, and how informative the decompositions are. We find that users significantly prefer our method to the baselines for both relevance to the input images as well as informativeness, further validating the interpretability of our method. Please see our general comment for more information, where we elaborate on the setup and results of this evaluation.

Negative concept weight performance. Thank you for the observation. As part of our desiderata for interpretability, we impose a nonnegativity constraint on the concept weights, as user studies (including our own) highlight that humans find negative concepts and weights to be confusing and unintuitive. In our optimization problem (2), we try to find a set of weights that minimize our reconstruction error. Constraining this set of weights to be nonnegative will reduce the search space and thus result in a worse reconstruction. So, we see in Table 5 that removing the nonnegativity constraint results in a more accurate reconstruction, but we lose the interpretability of our decompositions.

We again thank you for your constructive feedback!

[A] Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure.

Thank you for your comments! We appreciate your helpful feedback and address your concerns below. We will also correct the typo you mentioned.

Efficacy of task-agnostic concept sets. As requested by the reviewer, we include in the additional results concept decompositions of randomly chosen samples from Waterbirds and CelebA (Fig. 3b). We find that the decompositions include fine-grained and detailed concepts, such as the celebrity “Rihanna”, the brand “adidas” (in reference to a branded Adidas sweatband), and the species “pelican”. This shows qualitatively that our concept set is indeed task agnostic or at least broadly applicable to a variety of tasks. While we only include four in the additional rebuttal documents due to space constraints, we will include a more thorough set of examples in the final version. We also include quantitative results for a broader set of tasks from the original CLIP paper [1] in our Additional Materials (Table 1), as suggested by reviewer djnP. These results further indicate the wide applicability of our selected concept set. If the reviewer believes that the term “task-agnostic” is still inappropriate for this method, we are happy to change our language accordingly.

Adding class labels to the concept set. Thank you for this comment. For ImageNet, we find that the class names are often fine-grained animal species that are more than two words long (such as “European fire salamander”, “sulphur-crested cockatoo”, “red-backed sandpiper”, etc), making it difficult for one- and two-word concepts to capture. As such, these concepts can be added to the LAION dictionary to allow for full reconstruction, while maintaining the interpretability of all other concepts in the image. We include this augmented vocabulary in our experiments to demonstrate the efficacy of our sparse decomposition method, even if it highlights a limitation of the LAION vocabulary.

We also note that many images, including those in ImageNet, can be complex and contain semantic information outside of the class object. Even if the top concept is the class label, the rest of the decomposition may contain information relevant for prediction, especially if there are spurious correlations. Decompositions should include all of this information to allow for full understanding of the semantic content of the image, and more importantly to allow for use cases outside of prediction, such as editing. If you take the example of the Waterbirds dataset, we expect our decompositions to include both the class label and the land/water background concepts. Being able to edit the background concepts while maintaining the class label is useful for improved classification. Furthermore, this semantic information can be useful for other applications, such as dataset summarization, image tagging for retrieval, and exploration of unlabelled data. In summary, concept decompositions that include a combination of class concepts and other semantic information provide utility in a variety of settings beyond prediction; however, we will further clarify this in our final paper.

User study. We agree that user studies are important for assessing the human interpretability of concept based explanations. We have included results from a small-scale user study similar to that suggested by the reviewer in [23] (Task 2) in Figure 1 of the additional materials, where we present users with an image and two concept decompositions/explanations and ask the following questions:

• Which explanation is more relevant to the provided image?
• Which explanation is more relevant to the model’s prediction?
• Which explanation is more informative?

We benchmark against two other CLIP-interpretability methods: LF-CBMs [23] and IP-OMP [22]. We find that users significantly preferred SpLiCE to LF-CBMs for both (1) and (3), and significantly preferred SpLiCE to IP-OMP for (1, 2, 3). We also highlight that our method is able to produce similar/better concept decompositions, in terms of human interpretability, than the baselines without the need for class labels for concept mining and without training a classification probe, both of which are computationally expensive. For more discussion, please see the general rebuttal.

Decomposition for correct/incorrect samples. Thank you for the interesting question. In the additional material, Figure 3a, we include example SpLiCE decompositions for correctly and incorrectly classified samples using zero-shot with SpLiCE. We see that incorrect sample decompositions often contain correlated but slightly incorrect concepts that apply to other classes (“white dog” for an image of a white wolf) or concepts present in the image but not relevant to prediction (“baby toys” and “toddlers” for “abacus”). The former presents an instance where a practitioner might consider intervening on their predictor to improve performance, while the latter is simply an example of a difficult sample due to the noisiness of the image. In both cases, the explanation is useful for understanding the model’s reasoning and mistakes and can be used to intervene and improve performance.

Limitations of LAION-based concept set. This is an important limitation of our work. It is correct that limitations and biases in the dictionary will be reflected in downstream performance. We aimed to construct our dictionary to be as comprehensive as possible by using the training set of the model itself (as LAION was used to train Open-CLIP, the model used in this paper), so we are likely to include all concepts learned by the model. We also filtered out any unsafe or NSFW samples from LAION before constructing the dataset to limit harmful content. However, we acknowledge that this is a limitation and will be sure to make this point more clear in the final version. For more discussion please see the general comment.

We again thank you for your feedback and hope these comments will encourage you to reconsider your score.

Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.

Assumptions not justified. Thank you for this comment. Regarding Assumption 2, it is true that our experiments demonstrate the presence of non-semantic concepts in our decomposition. We clarify this point further in our general rebuttal, but our intent with Section 3 and our listed assumptions was not to claim that CLIP always satisfies these assumptions, but instead to reason from first principles regarding which mathematical properties enable vision-language models to have sparse decompositions. We plan to clarify this, move our discussion of these assumptions to the Appendix, and highlight our empirical investigations of CLIP’s semantic linearity (Assumption 3, experiments in Appendix B.5.) to the main paper, as suggested by reviewer djnP. We direct the reviewer to our general comment for further discussion of this point.

“Task-agnostic and concept datasets, training, or qualitative analysis”. Thank you for bringing this to our attention. We will be sure to update and relax our language as suggested, but we would also like to clarify our intent with this phrase here. We say that our method does not require concept datasets, in that traditional Concept Bottleneck Models (along with other methods such as tCAV) require training data that has both concept and class labels to train concept probes, whereas our method does not require a task and concept labeled image dataset. Instead, we construct our dictionary by parsing the text captions of LAION. As such, our method can be applied out of the box for a variety of tasks, without having to collect image datasets for each concept or task.

Second, while we do need to optimize for our concept weights, we do not need to train any concept probes or classifiers and accordingly do not need any training data, as SpLiCE decompositions perform sufficiently in zero-shot applications. This is where our work differs from many other similar works, such as [22, 23], which require training a sparse linear layer to obtain both explanations and predictions. This also permits SpLiCE to be used post-hoc and in low-data regimes.

Third, our point here was to say that we do not require qualitative analysis to label dictionary elements and thus generate explanations. More specifically, in prior work such as Sparse Autoencoders or neuron analysis in mechanistic interpretability research, dictionary elements are learned, uninterpretable vectors. As such, they require additional analysis to understand the semantics contained by each dictionary element, such as the auto-interpretation done in SAE literature, which involves decomposing large quantities of data, measuring correlations in the decompositions, and then qualitatively describing what each atom encodes. One of the benefits of SpLiCE is that we fix our concepts a priori rather than having to learn and then manually label them post-hoc which can result in errors. However, you are correct that we can use our resulting concept scores in downstream qualitative analysis by inspecting them and visualizing them.

For a detailed discussion of task-agnosticity, please see our general comment.

Related work. The connection you draw between our work and CBMs is very insightful. Our intent in claiming that we are not simply creating a concept-bottleneck model is to distinguish ourselves from works that require training classifier probes and that only consider the predictive case, as noted by the reviewer. We believe that this is a significant contribution of our work, as CLIP is frequently used in many non-predictive settings (retrieval, generation, etc), and because these decompositions allow for dataset summarization for data that is not labeled or does not have an associated task (e.g. unstructured web-scraped data). We agree that our work creates a concept bottleneck to represent CLIP embeddings, and we will discuss this further in the final version. Similarly, we agree that the vocabulary construction process is overseen by the authors, but it is not “supervised” in the traditional ML sense of the word, in that there are no labels or tasks it is optimized for. We also attempted to impose as minimal oversight and constraints on the dictionary as possible, simply ensuring that it was human-interpretable by using only 1- and 2- word concepts and safe by filtering NSFW content. Finally, we thank the reviewer for the provided citation and kindly note that we compare our work to the follow-up work by Fel et al, [31], in our submitted paper, noting that methods such as CRAFT and traditional NMF require feature visualization to understand each concept. We will be sure to clarify this comparison further.

Residual Concepts. This is an interesting question and future direction! Given that our dictionary has 15000 elements in 512 dimensions, our problem is very overcomplete, so it is reasonable to assume we can reconstruct CLIP embeddings with little loss even under a sparsity constraint. One way we could account for this residual error is to add it back into our decompositions and return the original CLIP representations. However, doing so results in a component of the representations being unexplained. For example, 98% of an embedding may be explained by our sparse decomposition, but there is a risk that the 2% residual which remains unexplained contains important information for downstream tasks. Depending on the application, one may wish to incur this loss to ensure that all information remains interpretable, or they may add the residual back in for applications that require perfect performance. We find this discussion to be similar in nature to the comparisons between post-hoc explanations and inherently interpretable models, and we believe the choice should be left to the user. We will include this discussion in our final version. Thank you!

Thank you for your feedback and taking the time during the discussion period to help improve our work! We hope to answer your remaining questions below.

Theoretical Analysis. Thank you for raising these questions regarding our theoretical analysis. As noted in the general rebuttal, we will attempt to address your concerns by moving much of the theoretical analysis section to the appendix and clearly describing its limitations and applicability in practice. We can also change our language regarding these assumptions in the Appendix and reframe them as hypotheses we have for why SpLiCE works. We appreciate your perspective on this section of the paper, and we believe your public discussion on OpenReview will be informative for future readers in the community.

CRAFT et al./NMF. We apologize for any confusion in our response, and if we are not understanding your question correctly. Our use of the term visualization may have been unclear. We are referring to the process of interpreting concept atoms by visualizing and manually analyzing examples that activate the atom, whether generated or from a dataset. We were not referring to visualizations created with an explanation method and summarizations of explanations, such as Fig. 4 of our paper.

We will focus specifically on NMF in the CRAFT paper to further elaborate on this. In CRAFT [A], the authors take image sets of examples and use NMF to learn a “concept bank” and set of coefficients. The resultant concept bank is a learned set of basis vectors, and therefore each element does not have any semantic or conceptual significance on its own. To label each atom in this concept bank (C1, C2, …), the paper states that they must “be interpreted by looking at crops that maximize the NMF coefficient” and consider “new sets of images containing [the concept]” (see captions of Fig. 2, 4 of [A]). By feature visualization, we refer to this process of labeling atoms by visualizing either test set exemplars and crops (as done in CRAFT and Sparse Autoencoder analysis) or synthetic, generated images (as is common in mechanistic interpretability neuron analysis) that highly activate each atom and describing these sets of images qualitatively [B, C (section “Manual Human Analysis”)]. This is a key difference between SpLiCE and CRAFT/NMF: for CRAFT/NMF, describing or interpreting dictionary atoms requires a qualitative summary of the images that activate said atom. In contrast, SpLiCE’s dictionary atoms are automatically interpretable because they inherently correspond to text phrases, thus not requiring this qualitative analysis. For more discussion of how our method relates to NMF and other dictionary learning methods, we kindly point the reviewer to our discussion with reviewer djnP. We hope this cleared up any confusion our terminology may have caused.

Thank you again for your engagement and support of the paper, and please let us know if you have any remaining questions!

[A] Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., ... & Serre, T. (2023). Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[B] Olah, C., Mordvintsev, A., & Schubert, L. (2017). Feature visualization. Distill

[C] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., ... & Olah, C. (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread