Thank you for your positive and valuable feedback. We address your concerns bellow. We will revise our paper according to your suggestions.

W1/Q1. The inference cost is significantly increased. / Comparison to existing CBMs on inference speed.

Thank you for this comment. We respectfully note that the direct comparison of inference speed between Z-CBMs and existing CBMs is unfair because the problem setting is different, i.e., existing CBMs require training. Even so, in order to identify the limitations of Z-CBMs and the level to which future research should aim, we compared the inference speed. We compared Z-CBMs with Label-free CBMs as follows.

Table 6-1. Evaluation on ImageNet

We can see that ZCBMs are at least twice as expensive to infer as Label-free CBMs. As the concepts contained in images are not known in the zero-shot problem setting, a relatively large value of $K$ to accommodate any concept for high accuracy needs to be selected. In this regard, future work could include approaches such as dynamically creating a cache of concepts according to task. We will add this discussion to the paper.

W2. On the relevance to “Visual Classification via Description from Large Language Models”

Thank you for sharing related work. The mentioned paper proposed a zero-shot classification method by LLM-generated text descriptions for each target class. In contrast to Z-CBMs, which directly decompose input features by concepts, this method makes a prediction based on the correlation between the input features and the task-specialized texts. In this regard, this method can also provide interpretability in a different way from Z-CBMs, but it requires (i) generating the task-specialized texts with LLM, (ii) restricting the inference algorithm to the CLIP style zero-shot classification, and (iii) calculating the contribution scores for each text independently. On the other hand, Z-CBMs can be used for arbitrary domains without external LLMs and arbitrary inference algorithms (e.g., training heads). Further, Z-CBMs can provide relative contribution scores among concepts by sparse regression, i.e., we can compare different concepts quantitatively. We will add the discussion to Sec. 5.

Q2. I’m wondering whether the visual and textual features are in the same space as shown in Fig 2 (a)

Thank you for this question. We show the PCA feature visualization in Fig.7 of the revised paper; we use PCA instead of t-SNE because PCA provides more realistic intuition in the feature space with computing only linear transformations. We see that, in reality, the clusters of visual and text features are separated. This known as the modality gap, is one of the challenges of CLIP representations [b]. Although mitigating the modality gap is out of the scope of our paper, Z-CBMs can alleviate the modality gap by interpreting input images by textual concepts, as seen in Fig. 7. We consider this is the reason why Z-CBMs improve the zero-shot CLIP in Table 1.

[b] Liang, Victor Weixin, et al. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." NeurIPS 2023.

Q3. Concepts such as “Not maltese dog terrier” cannot provide interpretable information for identifying categories

Thank you for this comment. Strictly speaking, we consider that the appropriate granularity of concepts depends on use cases and is undecidable in general. For example, if the user wants a fine-grained classification of dogs, "Not maltese dog terrier" might be useful information for assessing the decision boundary. If one wants to avoid concepts similar to target class names, one can control the appearance by modifying the threshold in the concept filtering (L225). Similarly, we can avoid specific sets of concepts by directly removing them from the concept bank. However, automatically controlling concepts and their granularity in general cases is an open question and should be resolved in future work.

Thank you for your knowledgeable review and many interesting suggestions. We address your concerns below. We are sorry for the multiple responses but hope that these responses help you to re-evaluate our work.

Table 5-1. Additional evaluation on ImageNet

MW1/Q1. Novelty is limited. How is this work different from [ConSe 2014]?

Thank you for providing related work. Although they are important and should be discussed, our work is totally different from theirs and has remarkable novelty. Here, we focus on the difference from [ConSe 2014], which is your primary concern. First of all, the zero-shot classification with [ConSe 2014] is completely different from that with Z-CBMs. ConSe infers a target label from a semantic embedding composed of a weighted sum of concepts of the single predicted ImageNet label, whereas Z-CBMs infer the label by the concept features retrieved from the concept bank and weighted by sparse regression. ConSe has three potential risks: (i) Since the zero-shot inference depends on the ImageNet label space, it cannot accurately predict target labels if there are no target-related labels in ImageNet, (ii) the prediction accuracy largely depends on the model's ImageNet accuracy, (iii) the cosine-similarity based prediction can produce concepts semantically overlapped with each other and thus the accuracy and interpretability is restricted. In contrast, our Z-CBMs directly decompose an input image feature into concepts via concept bank, so it is not restricted to any external fixed label spaces. Our concept regression can also provide semantically identical concepts via sparse regression algorithm. Furthermore, we compared the performance between ConSe and Z-CBMs; we implemented ConSe with pre-trained CLIP and concept bank, which were in the same setting as Z-CBMs. The results are shown in Table 5-1 and Table 1 of the revised paper. Our Z-CBMs largely outperformed ConSe, indicating that the methodology of Z-CBMs is superior to and different from ConSe. We will add ConSe as a zero-shot baseline and discuss it in the main paper.

The other works partially share some technical components with our work (e.g., sparse regression), but they differ greatly from our Z-CBMs: [Costa 2014] also depends on the existing classifier in the same fashion as ConSe, [Write 2013] requires additional training for seen classes, and [Object2Actions 2015] limits the domains to action recognition. More importantly, none of them focus on the interpretability of the models' decisions. In contrast, our work is the first work to build interpretable zero-shot concept bottleneck models from the combination of concept retrieval and concept regression using a large-scale concept bank without any additional training and restrictions of domains, as acknowledged the technical novelty by Reviewers xHwx and mgBg.

MW2/Q3. Weighting concept based on target classes

Thank you for this suggestion. We did not consider this direction because it does not achieve the goal of concept bottleneck models, i.e., predicting concepts from input, and then predicting final labels from the concepts. This goal is essential to provide the interpretability of the output. Even so, we evaluated this variant as shown in Table 5-1 (the row of "class-to-concept"). The variant of "class-to-concept" slightly decreased the zero-shot baseline performance. This may be because retrieving and weighting concepts from target class texts are not helpful in reducing the modality gap between image and text in contrast to Z-CBMs. Therefore, this direction does not seem promising in terms of either interpretability or performance.

MW3/Q3. Positive weight constraint on the linear regression would be beneficial

Since $F_{C_x}$ is defined in the real number space, the retrieved concept vectors also contain negative values. In such a sense, the linear regression requires the negative values of $W$ to reconstruct image features $f_\mathrm{V}(x)$ that are also defined in the real number space. In fact, when constraining the weights to be positive, the performance was significantly degraded ("Z-CBM w/ positive weight" in Table 5-1). Further, the negative concepts are helpful for a more detailed understanding of the decision boundary.

We appreciate your constructive and helpful comments and suggestions. We address your concerns below. We will revise our paper according to your suggestions.

W1/Q1. Additional experiments using concept bank with question-and-answer (Q&A) approach

Thank you for this valuable suggestion. We have shown in the GPT-3 (ImageNet Class) row of Table 5 an experiment using a concept bank (4K concepts) with a Q&A approach of Label-free CBM. In order to deeper understand the importance of the concept bank, we now compare it with a baseline utilizing the same concept bank. The results are as follows.

Table. 4-1 Evaluation on ImageNet with GPT-3 (ImageNet Class) Concepts

Our Z-CBMs achieved competitive performance even when using such a smaller concept bank. However, they degraded CLIP-Score. This suggests the importance of the concept bank. In our zero-shot setting, covering sufficient concept knowledge with an abundant vocabulary is important to map concept-to-label without learning. We will add this result and discussion to Sec. 4.6.

W2/Q2. Qualitative and quantitative concept comparisons of linear regression and lasso

We also thank this insightful suggestion. For qualitative evaluation, we add the concept visualizations of Z-CBMs with linear regression to Fig. 3 of the revised paper. We can see that linear regression tends to produce concepts that are related to each other. In fact, quantitatively, we also found that the averaged inner CLIP-Scores among the top-10 concepts of lasso is significantly lower than that of linear regression (0.6855 in lasso vs. 0.7826 in linear regression), which means that lasso produces more independent concepts than linear regression. These results emphasize the advantage of using sparse regression like lasso in concept regression to reduce redundancies of the concepts.

Q3. Does Fig. 6 represent the total time for a zero-shot inference? Provide a breakdown of the reported times.

Yes, the y-axis of Fig. 6 represents the total time. A detailed breakdown is as follows.

Table. 4-2 Execution time for zero-shot inference (milliseconds)

For any $K$, the computation time of concept regression is dominant for the total. This is a limitation of Z-CBMs: there is a trade-off between computation time and the accuracy of the zero-shot inference. Therefore, we will address resolving this trade-off and speeding up the zero-shot inference in future work. We will revise 4.6.3 to add this discussion.

Thank you for your positive review.

I can't find anything wrong with this paper except perhaps the lack of technical innovation. There is abundant literature on concept bottleneck models. Sparse regression on concept features is very widely used. Using retrieval to find relevant concepts is not technically interesting. In my opinion, this work does not add much value to the existing CBM literature

Thank you for mentioning this. We respectfully note that our paper has a technical innovation in addition to the existing CBMs. First of all, the idea of building CBMs for arbitrary vision-language models in a zero-shot manner is technically novel, as acknowledged by Reviewers xHwx and mgBg. Specifically, the design of Z-CBMs that search input-related concept candidates from large-scale concept banks and then predict the importance of each concept by sparse regression is not obvious and has significant novelty. Also, since this paper is the first work on zero-shot CBMs, it is important to solve the problem as simply as possible as a baseline for future work. To this end, designing a method with well-known technical components such as retrieval and sparse regression is reasonable. As you commented, simplicity is also a strength, so we would be happy if you look at the contributions that have made CBM possible with zero-shots, rather than the simplicity of individual techniques.

We appreciate your detailed and professional review and address your concerns below. We will revise our paper accordingly.

W1. Accuracy is not impressive, and the paper lacks analysis of the modality gap.

Thank you for this comment. We respectfully remark that our main contribution is building practical zero-shot interpretable models without target datasets or training (L078). Table 1 supports our practicality claim, as naive methods struggle to achieve competitive performance (Table 6). To analyze the modality gap, we followed [b] and measured L2 distances: $1.74\times 10^{-3}$ for image-to-label and $0.86\times 10^{-3}$ for concept-to-label features. This demonstrates that Z-CBMs significantly reduce the modality gap via concept regression. Additionally, Fig. 7 shows PCA feature visualizations, indicating that weighted concept sums effectively bridge image and text modalities. These findings highlight that our results are not "normal" and offer novel insights.

[b] Liang, Victor Weixin, et al. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." NeurIPS 2023.

W2. CLIP-Score may be misleading since it uses reconstructions from image features.

Apologies for any confusion. Table 2 presents averaged scores of the top-10 concepts, selected by sorting absolute regression coefficients without using reconstructed vectors from Eq. (3). We also did not weigh concepts by coefficients when computing CLIP-Score. Thus, the scores are not obvious, as they depend on the regression algorithm used. For example, lasso outperforms linear regression, indicating better concept selection (Table 2-1).

Table 2-1. Evaluations of Z-CBMs on ImageNet

W3/Q2. How was $\lambda$ tuned, and how does it affect results?

We selected $\lambda = 1.0\times10^{-5}$ by searching from ${10^{-2},10^{-3},\dots,10^{-8}}$, aiming for a non-zero concept ratio above 10% with $K=2048$ on a subset of ImageNet. This $\lambda$ was used consistently across all experiments. As shown in Fig. 8 of the revised paper, varying $\lambda$ affects concept sparsity and accuracy, underscoring the importance of careful $\lambda$ selection for balancing sparsity and performance.

W4. On the relevance to "Attributes2Classname (A2C)"

Thank you for sharing related work. Indeed, A2C shares ideas of predicting textual concepts (attributes) from images and then predicting the final class from the concepts. However, unlike A2C, which requires supervised training for both image-to-concept and concept-to-label mappings, our Z-CBMs operate without additional training or datasets.

W5. Z-CBMs rely on CLIP and cannot be used in incompatible domains

This is an inherited limitation of Z-CBMs and VLM-based CBMs (e.g., Label-free CBMs). Nevertheless, we already have domain-specific CLIP models like MedCLIP [c] and can easily expect such CLIP variants to appear in each domain. In such a sense, one advantage of Z-CBMs over other CBMs is the availability without learning when such foundation models emerge. Besides, giving interpretability to the foundation models through Z-CBMs will also serve as a baseline for building supervised CBMs, making a fundamental contribution to this area.

[c] Wang, Zifeng, et al. "Medclip: Contrastive learning from unpaired medical images and text." EMNLP 2022.

Q1. Can you provide more detailed comparisons to prior work with their concept sets?

Yes. We have already shown the Z-CBM's result of GPT-3 generated concepts of Label-free CBMs in Table 6. Here, we compare the performance when using the identical concept set:

Table. 2-2 Evaluation on ImageNet with GPT-3 (ImageNet Class) Concepts

Although this is not a fair comparison since the baselines learn concept-to-label mapping on supervised datasets while Z-CBMs do not, our Z-CBMs achieved competitive performance. However, the CLIP-Scores are lower than Z-CBMs (All) in Table 2. This suggests the importance of using a large-scale concept bank to accurately map concept-to-label without learning.

Q3. How well the method performs if using random matrix as concept features $F_{C_x}$?

We evaluated this case on ImageNet as follows:

Table 2-3. Top-1 Accuracy on ImageNet

Using random matrices gradually approach the zero-shot baseline performance (61.88) as $K$ increases. This is because larger numbers of random vectors have more chance to contain similar vectors to image features. However, real concepts of Z-CBMs always outperformed random concepts, demonstrating the advantage of using meaningful concepts.

We appreciate your constructive and insightful feedback. We address your concerns below. We will revise our paper according to your suggestions.

W1. Why does the proposed method improve the original CLIP? Is the modality gap really reduced?

Thank you for this valuable comment. Existing research [a] showed that converting the modality improves performance by reducing the modality gap. To further evaluate the modality gap, we compare the distances (modality gap) among images, weighted sums of concepts by Eq.(3), and ground-truth class prompt texts on the feature spaces with ImageNet samples by following [b]. The L2 distances were $1.74\times 10^{-3}$ in image-to-label and $0.86\times 10^{-3}$ in concept-to-label, demonstrating that there is indeed the modality gap, and Z-CBMs largely reduce it by representing images with textual concepts. Fig. 7 in the revised paper also shows the PCA feature visualizations, where the weighted concepts are located between image and text feature clusters, emphasizing the reduction of the modality gap. We will add this discussion with the reference and additional experiments to Sec. B.1.

[a] Qian, Qi et al. "Intra-modal proxy learning for zero-shot visual categorization with clip." NeurIPS 2023.

[b] Liang, Victor Weixin, et al. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." NeurIPS 2023.

W2. On the advantages of Z-CBMs compared to a simple interpretation of embedding.

One clear advantage of Z-CBM is that it ensures interpretability by guaranteeing that the final prediction is made from interpretable concepts. That is, we cannot ensure interpretability by just retrieving concepts because this does not guarantee that the concepts really contribute to the prediction. To ensure interpretability, our Z-CBMs estimate the importance of the concepts by concept regression and predict the final labels by actually using the weighted concept features. As you commented, intervention is also an advantage of Z-CBMs since it provides meaningful interpretations by changing parts of concepts. Further, the performance improvements by reducing the modality gap can be an advantage, as shown above.

W3. On the performance difference between linear regression and lasso

This is due to the unstable numerical computation of linear regression. If the feature dimension $d$ is smaller than the concept retrieval size $K$, the Gram matrix of $F_{C_x}$ in linear regression will be rank-deficient, i.e., there is no inverse matrix for the closed-form solution, and lasso can avoid this by sparse regularization. In our setting, we used $d=512$ and $K=2048$ as the default, so the unstable computation prevents the optimization from convergence. Even if this is not the case, the Gram matrix might be rank-deficient since concept retrieval can generate concepts that correlate with each other. We will add this explanation to Sec. 4.6.4.

Q1. Why was linear regression used instead of lasso in the intervention experiments?

This is because the intervened concepts are already sparse. In Sec. 4.5, we intervened the concepts with non-zero coefficients after sparse regression. Since the number of non-zero concepts is smaller than the feature dimension, we do not need to consider the unstable computation problem discussed above. If lasso is used here, it is inappropriate as an intervention experiment because the ground-truth concept does not influence the output when sparse regularization eliminates it.